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Preface 


While there has been a proliferation of high-quality books on the areas of combinatorics, 
algorithms, and optimization in recent years, especially now, there has remained a notable 
absence of a comprehensive reference work accessible to students that covers all essential 
aspects of these three fields, starting from their foundations. Our motivation for writing 
this book was to supply a clear and easily digestible resource designed specifically for 
traditional courses in combinatorial and algorithmic mathematics with optimization. 
These courses are typically pursued by students in mathematics, computer science, and 
engineering following their introductory calculus course. The book’s origins trace back to 
a series of lecture notes developed for courses offered at the University of Jordan and the 
Ohio State University. Its primary target audience is students studying discrete structures, 
combinatorics, algorithms, and optimization, but it also caters to scientists across diverse 
disciplines that incorporate algorithms. The book is crowned with modern optimization 
methodologies. Without the optimization part, the book can be used as a textbook in a one- 
or two-term undergraduate course in combinatorial and algorithmic mathematics. The 
optimization part can be used in a one-term high-level undergraduate course, or a low- to 
medium-level graduate course. 

The book is divided into four major parts, and each part is divided into three chapters. 
Part I is devoted to studying mathematical foundations. This includes mathematical logic 
and basic structures. This part is divided into three chapters (Chapters 1-3). In Chapter 
1, we study the propositional logic and the predicate logic. In Chapter 2, we study basic 
set-theoretic structures such as sets, relations, and functions. In Chapter 3, we study basic 
analytic and algebraic structures such as sequences, series, subspaces, convex structures, 
and polyhedra. 

Part II is devoted to studying combinatorial structures. Discrete mathematics is the study 
of countable structures. Combinatorics is an area of discrete mathematics that studies how 
to count these objects using various representations. Part II, which is divided into three 
chapters (Chapters 4-6), studies recursion techniques, counting methods, permutations, 
combinations, arrangements of objects and sets, and graphs. Specifically, Chapter 4 intro- 
duces graph basics and properties, Chapter 5 presents some recurrence-solving techniques, 
and Chapter 6 introduces some counting principles, permutations, and combinations. 

Part III is devoted to studying algorithmic mathematics, which is a branch of math- 
ematics that deals with the design and analysis of algorithms. Analyzing algorithms 
allows us to determine and express their efficiency. This part is divided into three chapters 


XV 


xvi 


Preface 


(Chapters 7-9). In Chapter 7, we discuss the asymptotic notations, one of the important 
tools in algorithmic analysis. Then we dive more into determining the computational com- 
plexity of various algorithms. In Chapter 8, we present and analyze standard integer, array 
and numeric algorithms. In Chapter 9, we present elementary combinatorial algorithms. 

Part IV is devoted to studying linear and conic optimization problems, and is divided into 
three chapters (Chapters 10-12). Chapter 10 introduces linear optimization and studies its 
geometry and duality. This chapter also studies simplex and non-simplex algorithms for 
linear optimization. Second-order cone programming is linear programming over vectors 
belonging to second-order cones. Semidefinite programming is linear programming over 
positive semidefinite matrices. One of the chief attractions of these conic optimization prob- 
lems is their diverse applications, many in engineering. In Chapter 11, we introduce second- 
order cone optimization applications and algorithms. In Chapter 12, we introduce semidef- 
inite optimization applications (especially combinatorial applications) and algorithms. 

Each chapter commences with a mini table of contents. This provides what readers can 
expect to find in the chapter. At the end of every chapter, readers will find a set of exercises 
aimed at applying their knowledge of the chapter’s concepts and enriching their under- 
standing. Solutions to all of these exercises can be found in Appendix A. The book is also 
accompanied by a companion website that connects students and instructors. This website 
includes binary and multiple choice questions together with their solutions. Additionally, 
to further aid readers, we have placed a list of references at the end of each chapter. 


Columbus, OH Baha Alzalg 
April 2022. 
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The term “logic” has a broad definition, and countless logics have been explored by 
philosophers, mathematicians, and computer scientists. For most individuals, “logic” typ- 
ically refers to either propositional logic or predicate logic. Propositional logic, a classical 
form, involves two truth values (true and false). Predicate logic, on the other hand, expands 
upon propositional logic by allowing explicit discussion of objects and their properties. 

In this chapter, we first study the propositional logic which is the simplest, and most 
abstract logic we can study. After that, we study the predicate logic. We start by introducing 
the notion of a proposition. 

Throughout this chapter (and the entire book), N = {1, 2, 3, ...} denotes the set of all nat- 
ural numbers, Z = {0, +1, +2, +3, ... } denotes the set of all integers, and Q = {a/b : a,b € 
Z and b 4 0} denotes the set of all rational numbers. For example, 2 and 3/4 are rational 
numbers, but V2 and z are irrational numbers. We also let R denote the set of all real 
(rational and irrational) numbers. 


1.1 Propositions 


In this section, we define a proposition and introduce two different types of propositions 
with examples. To begin with, we have the following definition. 
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1 Mathematical Logic 


Definition 1.1 A proposition is a statement that can be either true or false. 


It is essential to emphasize that the proposition must hold either a true or false value, and 
it cannot simultaneously possess both. We give the following example. 


Example 1.1 Identify each of the following statements as a proposition or not and provide 
any required clarifications. 


(a) 142=3. 

(b) 24+4=4. 

(c) Hillary Clinton is a former president of the United States. 

(d) Bill Clinton is a former president of the United States. 

(e) Be careful. 

(f) Is Abraham Lincoln the greatest president that the United States has ever had? 
(g) x+y =3. 


Solution 

(a) “1 +2 = 3” is a proposition (a true proposition). 

(b) “2+4 = 4” is a proposition (a false proposition). 

(c) “Hillary Clinton is a former president of the United States” is a proposition (a false 
proposition). 

(d) “Bill Clinton is a former president of the United States” is a proposition (a true propo- 
sition). 

(e) “Be careful” is not a proposition. 

(f) “Is Abraham Lincoln the greatest president that the United States has ever had?” is not 
a proposition. 

(g) “x+y = 3” is not a proposition. In fact, the values of the variables have not been 
assigned. So, we do not know the values of x and y, and hence it is neither true or 
false. a 


Sometimes a sentence does not provide enough information to determine whether it is 
true or false, so it is not a proposition. An example is, “Your answer to Question 13 is incor- 
rect.” The sentence does not tell us who we are talking about. If we identify the person, say 
“Adam’s answer to Question 13 is incorrect,” then the sentence becomes a proposition. 

Note that, for a given statement, being not able to decide whether it is true or false (due 
to the lack of information) is different from being not able to know how to verify whether it 
is true or false. Consider Goldbach’s conjecture!: “Every even integer greater than 2 can be 
written as the sum of two primes.”? Despite its origin dating back to 1742 and computational 
data suggesting its validity, this conjecture remains an open problem in number theory. It 
is impossible for this sentence to be true sometimes, and false at other times. As of now, no 
one has proved or disproved this claim, classifying it as a proposition with an undetermined 


1 Christian Goldbach (1690-1764) was a German mathematician who also studied law. He is remembered 
today for Goldbach’s conjecture, which is one of the oldest and best-known unsolved problems in number 
theory and all of mathematics. 

2 Prime numbers are those that are only divisible by one and themselves. For example, 7 is a prime 
number. 


1.1 Propositions 


truth value because it cannot simultaneously be both true and false, and its resolution may 
await future mathematical breakthroughs. 

There are some sentences that cannot be determined to be either true or false. For 
example, the sentence: “This statement is false.” This is not a proposition because we 
cannot decide whether it is true or false. In fact, if the sentence: “This statement is false” 
is true, then by its meaning it is false. On the other hand, if the sentence: “This statement 
is false” is false, then by its meaning it is true. Therefore, the sentence: “This statement is 
false” can have neither true nor false for its truth value. This type of sentence is referred to 
as paradoxes. The study of a paradox has played a fundamental role in the development 
of modern mathematical logic. 

Note that the sentence “This statement is false,” which is self-contradicting, is different 
from the sentence “This statement is true,” which is self-consistent on either choice. 
Nevertheless, neither is a proposition. The latter type of sentence is referred to as an anti- 
paradox.’ The question that arises now is: Why the sentence “This statement is true” is not 
a proposition? This question is left as an exercise for the reader. 

We now define atomic (or primitive) propositions. Intuitively, these are the set of smallest 
propositions. 


Definition 1.2 An atomic proposition is one whose truth or falsity does not depend on 
the truth or falsity of any other proposition. 


According to Definition 1.2, we find that all the propositions in items (a)-(d) of 
Example 1.1 are atomic. 


Definition 1.3 A compound proposition is a proposition that involves the assembly of 
multiple atomic propositions. 


The following connectives allow us to build up compound propositions: 


AND (A), OR(V), NOT (>), IF-THEN (>), IFF (e). 


Example 1.2 The following proposition is compound. 


If itisraining and you are outside, then you get wet. 
eee) — 


atomic prop. 1 A atomic prop. 2 atomic prop. 3 


a 


compound prop. 1 


VR 


compound prop. 2 


Note that 


e compound proposition 1 = (atomic proposition 1) A (atomic proposition 2), 
e compound proposition 2 = (compound proposition 1) > (atomic proposition 3). | 


3 A paradox is a logically self-contradictory statement that is not a proposition. 
4 An antiparadox is a self-supporting, self-validating statement that is not a proposition. 
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1 Mathematical Logic 


In the realm of natural language, such as English, ambiguity is a recurring challenge that 
writers and readers encounter. The following remark says that slight variations in context 
could drastically alter the intended message. 


Remark 1.1 Sentences in natural languages can often be ambiguous and words can have 
different meanings in the context in which they are used. 


To illustrate Remark 1.1, we consider the following sentences: 


e The sentence “You can download WhatsApp or Skype to ring friends” could mean that 
you can only download one of the applications or it could mean that you can download 
just one or download both. 

e The sentence “I smelled a chlorine-like odor and felt ill” implies that the odor of chlorine 
made you sick, but the sentence “I am majoring in CS and minoring in Math” does not 
imply that majoring in CS caused you to minor in Math. 

e The sentence “I went to Chicago and took a plane” could mean that you took a plane to 
travel to Chicago or it could mean that you went to Chicago and then took a plane from 
Chicago to another destination, such as Las Vegas. 


In mathematics and computer science, it is important to avoid ambiguity and for sen- 
tences to have a precise meaning. This is why people have invented artificial languages 
such as Java. 


1.1.1 Notations 


Rather than writing out propositions in full, we will abbreviate them by using propositional 
variables: P, Q, R,..., etc. 


Example1.3 (Example 1.2 revisited) 
In Example 1.2, let P be the proposition “It is raining,” Q be the proposition “You are 
outside,” and R be the proposition “You get wet.” Then we can abbreviate the compound 
proposition 

“If itis raining and you are outside, then you get wet” as PAQ >R. 


ee ees SS w= 
P A Q R a 


In Section 1.2, we will learn more about the connectives AND, OR, NOT, IF-THEN, and 
IFF. This will enable us to gain a comprehensive understanding of how these connectives 
function within the context of formal logic and the role they play in constructing logical 
statements and arguments. 


1.2 Logical Operators 


In this section, we study the following logical operators or connectives: Negation, conjunc- 
tion, disjunction, exclusive disjunction, implication, and double implication. 

Before starting with the logical operators, we introduce the truth tables which help us 
understand how such operators work by calculating all of the possible return values of 
atomic propositions. 


1.2 Logical Operators | 7 


Definition 1.4 A truth table is a mathematical table used to show the truth or falsity of 
a compound proposition depending on the truth or falsity of the atomic propositions from 
which it is constructed. 


Examples of truth tables will be seen very frequently throughout this chapter. 


1.2.1 Negation, Conjunction, and Disjunction 


This part is devoted to introducing the logical operators: Negation (NOT), conjunction 
(AND), disjunction (OR), and exclusive disjunction (XOR). 


1.2.1.1 Negation 
The logical operator “NOT” transforms a proposition into its opposite truth value via a 
negation. 


Definition 1.5 If P is an arbitrary proposition, then the negation of P is written =P 
(NOT P) which is true when P is false and is false when P is true. 


The truth table for “~=” is shown below. 


P aP 
T F 
F 


Example 1.4 If P is the proposition “It is raining,” then =P is the proposition “It is not 
raining.” E 


Remark 1.2 Negation does not mean opposite. For instance, if x is a real number and ~P 
is the proposition “x is not positive,” then you cannot conclude that =P is the proposition 
“x is negative” because x could be 0, which is neither positive nor negative. 


1.2.1.2 Double Negation 

In real numbers, two negative signs cancel each other out. Similarly, in propositional logic, 
two negations also cancel each other out. The double negation of P is —(-P) or =-P. The 
truth table for “~=” is shown below. 


P aP =P 
T F T 
F T F 


1.2.1.3 Logical Equivalence 
Logical equivalence is a type of relationship between two propositional formulas in propo- 
sitional logic. 
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Definition 1.6 When the truth values for two propositional formulas P and Q are the 
same, the propositional formulas are called logically equivalent, and this is denoted as 
P=QorP =} Q. 


For instance, from the truth table for “~~,” we have P = ~~=P. This equivalence is called the 
double negation law. 

The negation connective is unary as it only takes one argument. The upcoming connec- 
tives are binary as they take two arguments. 


1.2.1.4 Conjunction 
The logical operator “AND” connects two or more propositions via a conjunction. 


Definition 1.7 If P and Q are arbitrary propositions, then the conjunction of P and Q is 
written P A Q (P AND Q) which is true when both P and Q are true. 


The truth table for “A” is shown below. 


P Q PAQ 
T T T 
T F F 
F T F 
F F F 


Example 1.5 If P is the proposition “It is raining” and Q is the proposition “I have an 
umbrella,” then P A Q is the proposition “It is raining and I have an umbrella.” a 


1.2.1.5 Disjunction 
The logical operator “OR” connects two or more propositions via a disjunction. 


Definition 1.8 If P and Q are arbitrary propositions, then the disjunction of P and Q is 
written P V Q (P OR Q) which is true when either P is true, or Q is true, or both P and Q 


are true. 


The truth table for “v” is shown below. 


P Q PvQ 
T T T 
T F T 
F T T 
F F F 


Example 1.6 IfPis the proposition “I get married” and Q is the proposition “I live alone,” 
then P v Qis the proposition “I get married or I live alone.” a 


1.2 Logical Operators 


1.2.1.6 Exclusive Disjunction 

In English, the word “or” can be used in two different ways: inclusively or exclusively. For 
example, let us say that: a computer science course might require that a student be able 
to program in either Java or C++ before enrolling in the course. In this case, “or” is used 
inclusively: a student who can program in both Java and C++ would be eligible to take the 
course. On the other hand, for another example, let us say this: I get married or stay single. 
In this case, “or” is used exclusively: you can either get married or stay single, but not both. 


Definition 1.9 If P and Q are arbitrary propositions, then the exclusive-disjunction of P 
and Q is written P @ Q (P XOR Q) which is true when exactly one of P and Q is true and 
false otherwise. 


The truth table for “@” is shown below. 


P Q POQ 
T T F 
T F T 
F T T 
F F F 


1.2.2 Implication and Double Implication 


In this part, we study the following logical operators: implication (IF-THEN) and double 
implication (IFF). 


1.2.2.1 Implication 
The logical operator “IF-THEN” connects two or more propositions via an implication, thus 
forming a conditional proposition. 


Definition 1.10 Let P and Q be arbitrary propositions. The implication P implies Q (also 
called the conditional of P and Q) is written P —> Q (IF P THEN Q), which is false when P is 
true and Q is false, and true otherwise. Here, P is called the hypothesis and Q is called the 
conclusion. 


Example 1.7 IfPis the proposition “You live in Russia” and Q is the proposition “You live 
in the coldest country in the world,” then P > Q is the proposition “If you live in Russia, 
then you live in the coldest country in the world.” E 


We have five different ways to read P > Q, as is seen in the following remark. 


Remark 1.3 Let P and Q be two atomic propositions. The following statements have the 
same meaning as the conditional statement “If P then Q.” 


(a) QifP. (c) Qis necessary for P. 
(b) P is sufficient for Q. (d) P only if Q. 
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Example 1.8 Consider the implication: 
e Ifa person is a president of the United States, then s/he is at least 35 years old. 


According to Remark 1.3, the above implication can be restated in the following equiva- 
lent ways: 


e A person is at least 35 years old if s/he is president of the United States. 

e Being president of the United States is sufficient for being at least 35 years old. 

e Being at least 35 years old is necessary for being President of the United States. 

e A person is president of the United States only if s/he is at least 35 years old. a 


The truth table for “—” is shown below. 


P Q P>Q 
T T T 
T F F 
F T T 
F F T 


Therefore, the implication is true if we are on the lines (tt), (ft), or (ff) only, and it is false 
if we are on the line (tf). We now justify the truth table for “—.” Suppose that Sara is a 
math teacher and that she told her students in a class before the first midterm exam the 
statement that “If everyone in the class gets an A on the midterm, then I will make cookies 
for the class.” Note that this statement says nothing about what will happen if not everyone 
gets an A. So, if a student did not get an A, Sara is free to make cookies or not and she 
will have told the truth in either case. Thus, the only case where Sara did not tell the truth 
(i.e., the implication is false) is the case that if everyone got an A in the midterm (i.e., the 
hypothesis is true), but Sara did not make cookies for the class (i.e., the conclusion is false). 
This is the case that made the (tf)-line in the implication truth table. 


Example1.9 Let R denote the set of real numbers and x € R.° Decide whether each of 
the following implications is true or false. Justify your answer. 


(a) Ifx > 10, then x > 0. (d) Ifx? < 0, then x = 42. 

(b) Ifx > 0, then x > 10. (e) If the Riemann hypothesis is true, then 
(c) Ifx? > 0, then x = 42. xt > 0.6 

Solution 


The very first look tells us that the proposition in item (a) is true, and those in items (b) and 
(c) are false, but we might be uncertain about the propositions in items (d) and (e). Since all 
these propositions have the P > Q form, we shall analyze each item according to the truth 


5 To express the fact that an object x is a member of a set A, we write x € A (see Section 2.2 for definitions). 
6 The Riemann hypothesis states that all non-trivial zeros of a mathematical function, the Riemann zeta 
function, have a real part equal to 0.5. This hypothesis is still a mysterious unsolved problem of 
mathematics. 


1.2 Logical Operators 


table for “—.” As mentioned earlier, the conditional statement P > Q is true if we are on 
the lines (tt), (ft), or (ff) only, and it is false if we are on the line (tf). 


(a) Since the hypothesis is x > 10 and the conclusion is x > 0, we are 


(ff), ifx <0; 
on the line: < (ft), if0 <x < 10; 
(tt), if10<x. 


So, we cannot be on the (tf)-line for any x € R. Thus, the proposition is true. 
(b) Since the hypothesis is x > 0 and the conclusion is x > 10, we are 


(ff), ifx <0; 
on the line: < (tf), if0 <x < 10; 
(tt), if10<x. 


So, there is x € R that makes us on the (tf)-line. Thus, the proposition is false. 

(c) Take x = 2, then x? = 4 > 0, but x £ 42. So, we are on the (tf)-line. Hence, the proposi- 
tion is false. 

(d) Pis always false, so we cannot be on the (tf)-line. Hence, the proposition is true. 

(e) The Riemann hypothesis is an unsolved conjecture in mathematics. That is, no one 
knows if it is true or false at this time. But this does not prevent us from answering the 
question for this item. In fact, as Q is always true, we cannot be on the (tf)-line. Hence, 
the proposition is true. E 


Example 1.10 Usea truth table to show that P > Q = =P v Q. This equivalence is called 
the implication law. 


Solution 

We prove that two propositional formulas are logically equivalent by showing that their 
truth values are the same. A truth table that contains the truth values for P > Qand =P v Q 
is shown below and ends up the proof. 


P Q P>Q =P -PVQ 
T T T F T 
T F F F F 
F T T T T 
F F T T T 


1.2.2.2 Why Implications Are Important in Mathematics? 

Implications are important in mathematics because many mathematical theorems can 
be restated in the IF-THEN form, which enables us to prove them. See, for instance, the 
theorem that is stated and proved in the following example. 


Example 1.11 Prove the result of the following theorem. 


Theorem: The sum of two even integers is even. 
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Solution 
To prove the theorem, we restate it in IF-THEN form: 


Theorem Restatement: If x and y are both even integers, then x + y is even. 


Proof: If x and y are even, then each of x and y is the product of 2 and an integer. That is, 
x= 2n and y= 2m, for some n,m E Z, 


where Z denotes the set of integers. Then x + y = 2n + 2m = 2(n + m). Since the sum of 
any two integers is an integer, this means that x + y is the product of 2 and an integer, and 
hence is even. The proof is complete. E 


1.2.2.3 Contrapositive of an Implication 
The contrapositive of the implication P > Q is the implication ~Q > ~P. 


Example 1.12 The contrapositive of the proposition “If today is Monday, then Adam has 
a class today” is “If Adam does not have a class today, then today is not Monday.” a 


The truth table for the contrapositive is given below. 


T T F F T 
T F T F F 
F T F T T 
F F T T T 


From the truth table for contrapositive and that for the implication, we conclude that P > 
Q = ~Q > ~P. This is called the law of contrapositive. 
The contrapositive is important and useful in mathematics for two reasons: 


e Once a theorem in IF-THEN form is proven, there is no need to prove the contrapositive 
of the theorem. A proof by contrapositive, also known as contraposition, is a fundamental 
inference technique employed in proofs. It entails deriving a conditional statement from 
its contrapositive counterpart. 

e If we are trying to prove a theorem in IF-THEN form, sometimes it is easier to prove its 
contrapositive. 


Example 1.13 supports the first reason and Example 1.14 supports the second reason. 


Example 1.13 The Pythagorean theorem states that in any right triangle, the square of 
the length of the hypotenuse (the side opposite the right angle) is to equal the sum of the 
squares of the lengths of the legs of the right triangle. The proof of this theorem is beyond 
the scope of our discussion. 


1.2 Logical Operators 


In IF-THEN form, the Pythagorean theorem can be rewritten as: “If a triangle is right 
with hypotenuse c and legs a and b, then a? = b? + c’,” or equivalently: “If a triangle is 
right-angled, then its three sides satisfy a Pythagorean triple,” where a Pythagorean triple 
is a set of positive integers, a, b and c, that fits the rule a? = b? + c?. 

Since Pythagorean theorem was proven by Pythagoras, its contrapositive: “If triangle 
sides do not satisfy a Pythagorean triple, then the triangle is not right-angled” is obtained 
for free. a 


Example 1.14 Let x be a positive integer. Prove the result of the following theorem. 
Theorem: If x? is odd, then x is odd. 


Solution 
The implication in the theorem statement is equivalent to its contrapositive: 


If x is even, then x? is even, 


which is easier to prove. Let x be an even integer, then x can be written in the form x = 2k, 
for some positive integer k. It follows that x? = (2k)? = 2(2k?). Thus, x? is even. C] 


Note that a proof by contraposition is different from the so-called direct proof. In a 
direct proof, we write a sequence of statements which are either evident or evidently 
follow from previous statements, and whose last statement is the desired conclusion 
(the one to be proved). For instance, in Example 1.14, when we proved the statement that 
“If x is even, then x? is even” we used a direct proof. 


1.2.2.4 Converse of an Implication 
The converse of the implication P > Q is the implication Q —> P. The truth table for the 
converse is given below. 


P Q P>Q Q>P 
T T T T 
T F F T 
F T T F 
F F T T 


From the above truth table, it is clear that Q > P# P > Q. 


Example 1.15 The converse of the proposition “If today is Monday, then Adam has a class 
today” is “If Adam has a class today, then today is Monday.” Suppose that all Adam’s classes 
are on Mondays, Wednesdays, and Fridays though, then the original implication is true 
while its converse is false. This illustrates why we found that P > Q and Q > P are not 
logically equivalent. E 
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1.2.2.5 Inverse of an Implication 
The inverse of the implication P > Q is the implication ~P —> ~Q. The truth table for the 
inverse is given below. 


T T F F T 
T F F T T 
F T T F F 
F F T T T 


Itis clear that a conditional statement is not logically equivalent its inverse. That is, P > Q 
~P > ~Q. It is also clear, from the truth table for the inverse and that for converse, that the 
inverse and converse of any conditional statement are logically equivalent. That is, =P > 


=Q =Q >P. 


Example 1.16 The inverse of the proposition “If today is Monday, then Adam has a class 
today” is “If today is not Monday, then Adam does not have a class today.” E 


We now give the last logical operator, which is the double implication (IFF, which is read 
“if and only if”). 


1.2.2.6 Double Implication 
The following definition defines a double implication as the combination of an implication 
and its converse. 


Definition 1.11 Let P and Q be arbitrary propositions. The double implication (also- 
called the biconditional) of P and Q is written P - Q (P IFF Q), which is true precisely 
when either P and Q are both true or P and Q are both false. 


Example 1.17 IfP is the proposition “You live in Russia” and Q is the proposition “You 
live in the coldest country in the world,” then P + Q is the proposition “You live in Russia 


iff you live in the coldest country in the world.” E 


The truth table for “>” is given below. 


P Q PQ 
T T T 
T F F 
F T F 
F F T 


Note that P ~ Q means that P is both necessary and sufficient for Q. One of the chapter 
exercises asks to prove that P+ Q = (P > Q) A (Q > P). 


1.3 Propositional Formulas 


Table 1.1 A truth table for (P A Q) e ~R. 


P Q R aR PAQ PQ eR 
T T T F T F 
T T F T T T 
T F T F F T 
T F F T F F 
F T T F F T 
F T F T F F 
F F T F F T 
F F F T F F 


To prove an “if and only if” statement, you have to prove two directions. To disprove an 
“if and only if” statement, you only have to disprove one of the two directions. For instance, 
letting x be an integer, to prove that x? = 0 iff x = 0, we have to prove the two directions: 
First, we prove that if x? = 0 then x = 0. And second, we prove that if x = 0 then x? = 0 
(the proofs are trivial). To disprove x? = 4 iff x = 2, we only disprove the direction: If x? = 4 
then x = 2 (the disproof is trivial: take x = —2). 


Example 1.18 Construct a truth table for the compound proposition 


PAQ AR. 


Solution 

Since we have three propositional variables, P,Q, and R, the number of rows in our 
truth table is eight rows (see Remark 1.4). The truth table for (P A Q) + ~R is given in 
Table 1.1. E 


We end this section with the following remark. 


Remark 1.4 The number of rows in a truth table for a propositional formula with n vari- 
ables is 2” rows. 


For instance, the numbers of rows in the truth tables for the propositional formulas 
=P, PA Qand (PA Q) © ~R are 2}, 2”, and 2? rows, respectively. 


1.3 Propositional Formulas 


In this section, we study special propositional formulas, which are tautologies, contradic- 
tions, and contingencies. Then we study how to negate compound propositions and show 
how to derive propositional formulas. Before we dive into these, we illustrate the order in 
which the logical operators will be applied. 
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Table 1.2 The precedence order of logical operations. 


Operation(s) Operator(s) Precedence 
Parentheses () 1 
Negation (NOT) = 2 
Conjunction (AND) A 3 
Disjunction (OR, XOR) V,® 4 
Implication (IF-THEN) > > 
Double implication (IFF) < 6 


1.3.1 Order of Logical Operations 


In arithmetic, multiplication has higher precedence than addition. Hence, the value of the 
expression 4 + 5 x 2is not 18, but 14. Similarly, in propositional logic, logical operators have 
operator precedence the same as arithmetic operators. We preserve the precedence order 
specified in Table 1.2. 


Example 1.19 Determine the value of the propositional formula P v Q A R if the propo- 
sitional variables P, Q, and R have values of false, true and false, respectively. 


Solution 

From Table 1.2, the operator A has higher precedence than the operator v. So PV QAR = 
P v (Q A R). Plugging in the values of the variables, we have F v (T A F), which has the value 
of false. E 


Note that, for longer expressions, we recommend using parentheses to group expressions 
and control which ones are evaluated first. For example, the propositional formula P V QA 
R > PV Ris written as (PV (QAR)) > (PV R), and the propositional formula ~P v Q = 
R > S is written as ((-P) v Q) = (R > S). Note also that if we have two logical operators 
with the same precedence, then their associativity is from left to right. For example, P ® 
Q v Ris written as (P ® Q) V R, and P > Q > Ris written as (P > Q) > R. 


1.3.2 Tautologies, Contradictions, and Contingencies 

Tautologies, contradictions, and contingencies are propositional formulas of particular 
interest in propositional logic. We have the following definition. 

Definition 1.12 A propositional formula is called 


(a) a tautology if it is always true, 
(b) a contradiction if it is always false, 
(c) a contingency if it is neither a tautology nor a contradiction. 


Let P be an arbitrary proposition. From Table 1.3, it is seen that P v =P is a tautology, 
P ^ =P is a contradiction, and P > (P A =P) is a contingency. 


1.3 Propositional Formulas 


Table 1.3 A tautology, contradiction, and contingency. 


P =P Pv-P Pa-P P > (P A =P) 
T F T F F 
F T T F T 


Table 1.4 A truth table for the propositional formulas of Example 1.20. 


P Q PAQ Pv@Q (PAQ)>P (PvQ)>P 
T T T T T T 
T F F T T T 
F T F T T F 
F F F F T T 


Example 1.20 Use a truth table to determine if each of the following propositional for- 
mulas is a tautology, contradiction, or contingency. 


(a) PAQ >P. (c) (PV Q) > (AP V Q. 
(b) (PY Q >P. 


Solution 
From Table 1.4, it is seen that (P A Q) > Pisa tautology and (P v Q) > P is a contingency. 
This answers items (a) and (b). Item (c) is left as an exercise for the reader. a 


Last, but not least, itis worth mentioning that Definition 1.12 suggests an alternative defi- 
nition to the notion of logical equivalence: Two propositions P and Q are logically equivalent 
if P © Qis a tautology. 

A proof by contradiction is a common technique of proving mathematical statements. In 
a proof by contradiction, to prove that a statement P is true, we begin by assuming that P is 
false, and show that this assumption leads to a contradiction, then this contradiction tells 
us that our original assumption is false, and hence the statement P is true. The following 
example illustrates how this proof technique is used. 


Example 1.21 Prove that V2 is an irrational number. 


Solution 

Suppose, in the contrary, that V2 is rational. It follows that there are two integers, say a 
and b, such that V2 = a/b. Without loss of generality, we can assume that a and b have no 
common factors (otherwise, we can reduce the fraction and write it in its simplest form). 
Multiplying both sides of the equation V2 = a/b by b and squaring, we get a? = 2b*. This 
means that a? is even. From the theorem stated in Example 1.14, a itself must be even. 
Thus, a = 2m for some integer m. It follows that 


2b? = a? = (2m)? = 4m?, 
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and hence, after dividing by 2, we have b? = 2m?. This means that b?. By the same theorem 
stated in Example 1.14, b itself must be even. We have shown that both a and b are even, and 
hence they are both multiples of 2. This contradicts the fact that a and b have no common 
factors. This contradiction tells us that our original assumption is false, and hence V2 must 
not be rational. Thus, V2 is irrational. The proof is complete. a 


1.3.3 Negating Compound Propositions 


In this part, we study how to negate negations, conjunctions, disjunctions, and implications. 


1.3.3.1 Negating Negations 

When we say “Results of the numerical experiment are not inconclusive” means “Results of 
the numerical experiment are conclusive.” So, if you negate a negation they effectively can- 
cel each other out. This leads us to the double negation law =-P = P, which was previously 
stated in this chapter. 


1.3.3.2 Negating Conjunctions 

Suppose your roommate made the statement “I will get an A in Software I and an A in 
Foundations I.” If your roommate’s prediction is not correct, then s/he did not get an A in 
Software I or did not get an A in Foundations I. This leads us to the first DeMorgan’s law: 


(PAQ) = AP Vv -=Q. 


Table 1.5 verifies the first DeMorgan’s law. 


1.3.3.3 Negating Disjunctions 

Suppose your roommate made the statement “I will get an A in Software I or an A in 
Foundations I.” If your roommate’s prediction is not correct, then s/he did not get an A 
in Software I and did not get an A in Foundations I. This leads us to the second DeMorgan’s 
law: 


A(P VQ) = ~P AQ. 


Constructing a truth table to verify the second DeMorgan’s law is left as an exercise for 
the reader. Note that DeMorgan’s laws can be extended to expressions with more than one 
conjunction and disjunction as follows: 


aAPAQAR)=7PV-7AQV AR, APVQVR)=APATQARR. 


Table 1.5 A truth table verifying the first DeMorgan’s law. 


P Q =P -Q PAQ (PAQ) =P v ~Q 
T T F F T F F 
T F F T F T T 
F T T F F T T 
F F T T F T T 


1.3 Propositional Formulas 


Example 1.22 Negate and simplify the following two compound propositions. 
(a) (PV QARR. (b) (PV Q)V (RAP). 


Solution 
(a) (P V Q) A aR) = “(PV Q) V œR (DeMorgan’s law) 


= ~(PvQ)vVR (Double negation law) 


= (-PA7Q)VR. (DeMorgan’s law). 
(b) This item is left as an exercise for the reader. E 


1.3.3.4 Negating Implications 

Suppose your (rich) roommate who is stingy made the request that “If you go to the super- 
market, buy him organic food with your money.” Then you realized that it is time to teach 
him an unforgettable lesson for having these miserly attitudes and asking you this. What 
should you do? Is the correct answer by not going to the supermarket and not buying him 
organic food? No. In fact, the best way is “You go to the supermarket and you do not buy 
him organic food.” This leads us to the law 


(P > Q) = PAQ, (1.1) 
which we call the implication negation law. The logical equivalence in (1.1) immediately 


follows by noting that 


~(P > Q) = =C Pv Q) (Implication law) 
(a-P) A7=Q_ (DeMorgan’s law) 


= PAQ. (Double negation law) 


We can negate and simplify conditional statements either by directly applying the impli- 
cation negation law, such as 


(PAQ) > R) = PAQARR, (1.2) 
or by applying earlier logical equivalences, such as 


~(P > (QV R) = PA7(QVR)_ (Implication negation law) (1.3) 
= PA7AQA-R. (DeMorgan’s law) ` 
In Exercise 1.12, we learn how to negate exclusive disjunctions and double implications. 


1.3.3.5 Universal Sets 
We have seen that we can transform any propositional formula into an equivalent formula 
in disjunctive normal form (DNF) that uses only operators from the set {A, V, =}. 


Definition 1.13 A set of operators, such as {A,V,7}, that can be used to express any 
proposition is called a universal set. 


Note that the set of operators { ^, =} is universal. To see this, it is noted the proposition P v Q 
can be expressed using just ~ and A. In fact, using double negation and DeMorgan’s laws, 
we have 

PVQ=--(PV Q) = 7(-P anQ). 


So, we can for instance write P V (~Q A R) = a[ ~P Aag A R)]. 
Note also that the set of operators {v, =} is universal, while {v, A} is not. 
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1.3.4 Modeling Using Propositional Logic 


Propositional logic modeling is the process of abstracting a propositional logic problem from 
the real world. We have the following example. 


Example 1.23 


(a) Model the following statement using propositional logic tools. 
“Tf it is raining and Paul does not have an umbrella, then he will get wet.” 

(b) Simplify the contrapositive of the propositional formula obtained in item (a), then trans- 
late it back to English. 


Solution 
(a) Let P, Q, and R be three propositional variables defined below. 
P -Q R 
mm —e mm 
If it is raining and Paul does not have an umbrella, then he will get wet. 


(Here, Q is the proposition “Paul has an umbrella”). Therefore, the statement can be 
symbolized as (P A =Q) > R. 

(b) The contrapositive of the propositional formula (P A ~Q) > R is the propositional 
formula ~R > ~(PA7Q). Using the DeMorgan’s and double negation laws, this 
formula can be simplified to ~R > (~P v Q). Translating back to English, we get 


If Paul did not get wet, then it was not raining or he had an umbrella. 
\ j \ fp aeea amaaa 
-R =P Q a 
The general scheme for solving a logical word problem is as follows. We construct a propo- 
sitional logical model from a logical word problem statement in a problem domain. The 
model is then solved to obtain a logical conclusion, which can be finally translated back 
into the problem domain. See Figure 1.1. 
We have the following examples. 


Example 1.24 Negate the following statements. 


(a) If Sara gets an A in Foundations I and COVID-19 disappears, then she will travel to San 
Francisco. 
(b) If COVID-19 disappears, then Adam will travel to San Francisco or Los Angeles. 


Figure 1.1 Conceptual relationships 
among problem domain, modeling, and 
solution strategies. 


Problem domain 


Logical model Logical result 
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Table 1.6 A truth table verifying that the conditional statement in (1.4) is a tautology. 


aALA(NVM)A 
L M N aL NvM M> L (M > L) (1.4) 
T T T F T T F T 
T T F F T T F T 
T F T F T T F T 
T F F F F T F T 
F T T T T F F T 
F T F T T F F T 
F F T T T T T T 
F F F T F T F T 
Solution 


(a) This statement can be symbolized as the propositional formula (P A Q) > R, where P 
is “Sara gets an A in Foundations I,” Q is “COVID-19 disappears” and R is “Sara will 
travel to San Francisco.” According the equivalence (1.2), the negation is PA QA 7R. 
Translating back to English, we get the statement “Sara got an A in Foundations I, 
COVID-19 disappeared and she did not travel to San Francisco,” which is the negation 
of the original statement. 

(b) Let P be the proposition “COVID-19 disappears,” Q be the proposition “Adam will travel 
to San Francisco,” and R be the proposition “Adam will travel to Los Angeles.” The given 
statement can then be represented as the propositional formula P —> (Q v R), which, 
using the logical equivalence (1.3), has the negation P A ~Q A ~R. Translating back to 
English, we get the statement “COVID-19 disappeared and Adam did not travel to San 
Francisco nor to Los Angeles,” which is the negation of the original statement. E 


Example 1.25 Consider the following propositional logic word problem: 


Laila is not a Lebanese. 
Problem statement: <4 Nora is a New Zealander or Maria is a Macedonian. 
If Maria is a Macedonian then Laila is a Lebanese. 


Can you conclude that Nora is a New Zealander? Justify your reasoning and conclusion by 
linking them to the propositional logic. 


Solution 
Let L be “Laila is a Lebanese,” M be “Maria is a Macedonian,” and N be “Nora is a New 
Zealander.” Then the problem statement can be formulated in the following propositional 
logical model. 
aL, 
Problem model: <4 N V M, 
M >L. 
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The desired conclusion is an affirmative answer to the following question. 
Is the propositional formula [(=L) A (N V M) A (M > L)] —— N a tautology? (1.4) 


Otherwise, we cannot conclude that Nora is indeed a New Zealander. It is seen from 
Table 1.6 that Nora is a New Zealander. E 


1.3.4.1 Propositional Formulas in Computer Programs 

Propositions and logical connectives arise all the time in computer programs. As an 
example, consider the code snippet given” in Algorithm 1.1. In Algorithm 1.1, || denotes 
the logical OR operator and && denotes the logical AND operator. The condition in the 
“if-statement” can be simplified. Letting P be “x > 0” and Q be “y > 100,” then the con- 
dition can be symbolized as P v (=P A Q). In Example 1.26(a) (see also Example 1.27(a)), 
we will prove that P v (-P A Q) = P v Q. This means that we can simplify the code snippet 
without changing the program’s behavior. See Algorithm 1.2. Simplifying expressions in 
software can increase the speed of your program. 


1.3.5 Deriving Logical Equivalences 


We can derive (and prove) logical equivalences either by constructing corresponding truth 
tables or by applying earlier logical equivalences. 


1.3.5.1 Using Truth Tables to Prove Logical Equivalences 
If the truth values of two propositional formulas are identical under all possible interpreta- 
tions,® then they are logically equivalent. 


Example 1.26 Prove or disprove each of the following logical equivalences.? 


(a) PV(APAQ)=PVvQ. (b) PV(Q@R) = (PVQ) @ (PVR). 


Solution 
From the last two columns of Table 1.7, it is seen that the truth values of the propositional 
formulas P V (=P A Q) and P v Q are identical under all interpretations, hence they are 


Table 1.7 A truth table verifying PV (APA Q =Pv Q. 


P Q =P —PAQ Pv (=P AQ) PvQ 
T T F F T T 
T F F F T T 
F T T T T T 
F F T F F F 


7 “If-statement” written in the code is formally introduced in Section 7.1.1. 

8 An interpretation assigns a truth value to each propositional variable. 

9 “Prove or disprove” means that either you choose to give a proof that the given statement is correct or you 
give a counterexample (assignments for the variables P, Q, ...) that shows the given statement is incorrect. 
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logically equivalent. This proves the desired result in item (a). Item (b) is left for the reader 
an exercise. a 


Algorithm 1.1: A code snippet 


1: if (x > 0 || (Œ < 0 && y > 100)) then 
2: : 


3: | (further instructions) 


4: end 


Algorithm 1.2: The code snippet in Algorithm 1.1 revisited 


1: if (x > O||y > 100) then 
2: : 


3: | (further instructions) 


4: end 


1.3.5.2 Applying Earlier Equivalences to prove logical equivalences 

Recall that a direct proof is a sequence of statements which are either givens or deduc- 
tions from previous statements, and whose last statement is the conclusion to be proved. 
In particular, we use a sequence of logical equivalences to prove the logical equivalence 
of two compound propositions. In propositional logic, many common logical equivalences 
exist and are often listed as laws or properties. In Table 1.8, we include a list of the most 
such common logical equivalences. Item (a) of Example 1.27 uses Table 1.8 to reverify the 
logical equivalence in item (a) of Example 1.26. 


Example 1.27 Use logical equivalences to show that: 
(a) PV(APAQ)=PVQ. (c) [PA(P > Q)] A 7Q is a contradiction. 
(b) (PA Q) > Pisa tautology. 
Solution 
We use direct proofs. The following sequence of logical equivalences proves the logical 
equivalence of P v (=P A Q) and Pv Q. 
PV (APAQ) = (PV7>P) A(PV Q) (Distributive law) 
T A(PV Q) (Tautology law) 
=PvQ (Identity law). 


This proves the desired result in item (a). For item (b), we prove that (PA Q) > Pisa 
tautology by showing that (P A Q) > P and T are logically equivalent, which is seen from 
the sequence of equivalences. 

(PAQ) >~P =-(PAQ)VP (Implication law) 
(=P V73Q) Vv P_ (DeMorgan’s law) 
=P v (aQ v P) (Associative law) 
=P v (PV 7=Q) (Commutative law) 
(AP V P)V-7=Q (Associative law) 
=TVv-7Q = T (Tautology and domination laws). 


This proves the desired result in item (b). Item (c) is left for the reader an exercise. a 
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Table 1.8 A list of the most common logical equivalences. 


Name 


Identity laws 


Domination laws 


Idempotent laws 


Tautology law 
Contradiction law 
Double negation law 


DeMorgan’s laws 


Contrapositive law 
Implication law 
Implication negation law 
Double implication law 


Commutative laws 


Associative laws 


Distributive laws 


Exclusive distributive laws 


Formula(s) 
PAT=P 
PVF=P 
PAF=F 
PvT=T 
PVP=P 
PAP=P 
PvV7AP=T 
PA7P=F 
P=-(—P) 


(PA Q) = -=PV nQ 
-A(P VQ) = ~P A7Q 


P>Q=-7Q--7P 
P>Q=-PvVQ 
=(P > Q) = PA-7Q 


PoQ=P-QAQ-P) 


PVQ=QVP 
PAQZ=QAP 
P®Q=QQP 


(PVQ)VR=PV(QVR) 
(PAQ)AR=PA(QAR) 
(PEQ)GR=POQPR) 
PV(QAR)=(PVQ)A(PVR) 
PA(QVR)=(PAQ)V (PAR) 
PA(QQ@®R)=(PAQVG(PAR) 


P@QS(PVQAWPAQ) 
P OQ = (PAQ) V (AP AQ) 


1.4 Logical Normal Forms 


From mathematics classes, we know that f(x) = x? defines a function that squares its one 
input, and g(x, y) = 2x + y defines a function that doubles its first input and adds the second 
input. The domain and codomain of such functions are real numbers. 


1.4 Logical Normal Forms 


Table 1.9 A truth table defining a Boolean 
function f(-,:,-) on three variables. 


P Q R F(P,Q,R) 
T T T F 
T T F T 
T F T F 
T F F T 
F T T F 
F T F F 
F F T T 
F F F F 


We can also define functions whose domain and codomain are {T, F}. Such functions 
are called Boolean functions.'° For example, Truth Table 1.9 defines a Boolean function 
fC.: :) on three variables. In this section, we learn how to derive a propositional formula 
for the Boolean function f(-, -,-) represented by Truth Table 1.9. In particular, the Boolean 
disjunctive and conjunctive normal forms are given. 


1.4.1 Disjunctive Normal Forms 


To present the logical normal forms, we first have some definitions. 


Definition 1.14 A literal is an atomic proposition such as P, Q, or R, or the negation of an 
atomic proposition such as ~P, =Q, or ~R. 


Definition 1.15 A conjunctive clause is the conjunction of one or more literals. 
For example, P, ~Q A Rand =P ^ Q A R A 2P are all conjunctive clauses. 


Definition 1.16 A proposition is in disjunctive normal form (DNF for short) if it is the 
disjunction of one or more conjunctive clauses. 


Informally, a proposition is in DNF if it is an ORing of AND clauses. For example, the 
propositions ~P, PA ~Q, ~Qv R, and (PA7Q) v =P v (P AQ A R) are all in DNF. In con- 
trast, the propositions ~(P v Q) and (P A ~Q) v (Pv QV R) are not in DNF. 

When we derive explicit propositional formulas for Boolean functions, such as f(., -, -) 
represented by Truth Table 1.9, we will see that the obtained formulas are in logical normal 
forms such as DNF. The two-step procedure in the following workflow, followed by two 
examples, will teach us how to create such propositional formulas. 


10 Named after George Boole, a 19th-century logician. 
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Workflow1.1 We create a propositional formula for a Boolean function defined by a 
truth table by following two steps: 


(i) Create a connective clause for each row where the value of the function is true. 
(ii) Create a disjunction of the conjunctive clauses obtained in Step (i). 


Example 1.28 Create a propositional formula using logical operators that is equivalent to 
the function f defined in Truth Table 1.9. 


Solution 

First, by creating a connective clause for each row where the value of the function is true, 

we get the conjunctive clauses shown in Table 1.10. Note that the clause P A Q A ~R, for 

instance, evaluates to false for all other values of P, Q, and R since it is a conjunction. 
Next, by creating a disjunction of the conjunctive clauses obtained above, we obtain the 

formula 


f(P,Q, R) = (PAQ ATR) V(PAAQ AAR) V (APA 7AQ AR), 
which is in DNF. E 


Example 1.29 Write a truth table for each of the following propositions and use it to find 
an equivalent proposition in DNF. 


(a) =P > Q). (b) (PAQ) VAR. 
Solution 
(a) A truth table for the proposition =(P —> Q) is given in Table 1.11. It is also seen from the 


table that =(P > Q) = P A ~Q which is the equivalent proposition in DNF. 
(b) A truth table for the proposition (P A Q) V ~R is given in Table 1.12. It is seen that 


(PA Q)V AR = (PAQAR)V (PA QATAR) V (PA7Q A AR) 
V(AP A QATR)V (AP A7Q A 77R), 


which is in DNF. a 


The three-step procedure in the following workflow, followed by an example, will learn 
us how to convert any proposition into DNF. 


Table 1.10 The truth table of Example 1.28. 


v 
1O 
o] 


f(P,Q,R) Conjunctive clauses 


T = PAQAR7R 


T = PA7AQAR7R 


T = aPAnQAR 


Pom tease a 
Poa tte A 
tad ema was 
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Table 1.11 The truth table of item (a) in Example 1.29. 


P Q P> Q -=(P > Q) Conjunctive clauses 
T T T F 

T F F T = PA-7Q 

F T T F 

F F T F 


Table 1.12 The truth table of item (b) in Example 1.29. 


P Q R PaQ aR (PAQ)vV-R Conjunctive clauses 
T E T T F T = PAQAR 

T T F T T T = PAQA-R 

T F T F F F 

T F F F T T = PA7QAA7R 

F T T F F F 

F T F F T T = “PAQAA7R 

F F T F F F 

F F F F T T = aPAnQ AR 


Workflow 1.2 We transform a proposition into DNF using a sequence of logical equiva- 
lences by following three steps: 
(i) Convert any symbols that are not v, A, or ~ using logical equivalence laws. For instance, 
P>Q=-7PVQ. 
(ii) Use the DeMorgan’s laws and the double negation law, as needed, so that any negation 
symbols are directly on the variables. 
(iii) Use the distributive, associative, and commutative laws, if needed. 


Example 1.30 Transform the proposition (P > (Q > R)) v ~(P v a(R v S)) into DNF. 


Solution 
The following sequence of logical equivalences transforms the given proposition into DNF. 


(P > (Q > R) VPV ARV S) 


= (mGPv (Q> R)vaPva(Rv S) (Implication law) 
= (“PV (AQV R)) V-7(P Vv (Rv S)) (Implication law) 
= (APV(AQV R)) V (AP A77(RV S)) (DeMorgan’s law) 
= (“PV (AQV R)) V (APA (RV S)) (Double negation law) 


(AP V (AQV R))V((APAR)V (AP AS)) (Distributive law) 
(AP) V (AQ) V RV (APAR)V (AP AS). “ 
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1.4.2 Conjunctive Normal Forms 
The conjunctive normal form (CNF) and the DNF can be viewed as the dual of each other. 


We have this definition. 


Definition 1.17 A proposition is in conjunctive normal form (CNF for short) if it is the 
conjunction of one or more disjunctive clauses, where each clause is a disjunction of one 
or more literals. 


Informally, a proposition is in CNF if it is an ANDing of OR clauses. For example, the 
following propositions are all in CNF: 
=P, ~Q V R, PA7Q, and (PV 7AQ)AAPA(PV QV R). 
In contrast, the following propositions are not in CNF: 
~(P v Q) and (PV AQ) A (PV QAR). 
The three-step procedure in the following workflow, followed by an example, will learn 
us how to convert any proposition into CNF. 
Workflow 1.3 We transform a propositional formula into CNF by following three steps: 


(i) Create a truth table for the negation of the proposition. 
(ii) Use the truth table from Step (i) to create an equivalent proposition in DNF. 
(iii) Negate the proposition from Step (ii), and apply DeMorgan’s law and the double nega- 
tion law to convert the proposition to CNF. 


Example 1.31 Convert the proposition (P A Q) V =R into CNF. 


Solution 
A truth table for the negation of the proposition (P A Q) V TR is given in Table 1.13. 
This shows that 


aA[(P A Q) VAR] = (PAAQAR)V APAQAR)V (AP A7Q AR), (1.5) 


Table 1.13 The truth table of Example 1.31. 


v 
ie) 
a 
J 
a 


PAQ (PAQ)v-=AR a[(P A Q) v AR] Conjunctive clauses 


F 


T = PA7AQAR 


T = mPAQAR 


T = APAAQAR 


7mm aa A 
c: ies e e e s e: a 
o e e: e e s e s: 
oe se oe oc > ie S S 
Ame metas 
Sma mtea ra RB 


1.5 The Boolean Satisfiability Problem 


which is in DNF. Then 


(PAQ VaR = ALP A Q)V mR] (DNL) 
= A[(PAAQAR)V (APA QAR) V (APA7AQ AR) By (1.5) 
= A(PAAQAR)AA(APA QAR) AAP AA7AQ AR) (DL) 
= (AP V ~Q VAR) A (A5PV AQ VAR) A (ATP V 77Q VAR) (DL) 
= OPV QVAR)A(PV AQV AR) A (PV QV AR) (DNL), 
where DNL and DL stand for the double negation law and DeMorgan’s law, respectively. 


1.5 The Boolean Satisfiability Problem 


The Boolean satisfiability is the problem of determining if a given propositional formula is 
satisfiable or not; see for example (Knuth, 2022, Subsection 7.2.2.2). We have the following 
definition. 


Definition 1.18 A propositional formula is said to be satisfiable if it evaluates to true for 
some values of its variables. If a propositional formula is not satisfiable, it is called unsatis- 
fiable. An unsatisfiable formula evaluates to false on any values of its variables. 


For example, the proposition (P V Q) > ~P is satisfiable because it evaluates to true when P 
is false and Q is true. Note that a satisfiable proposition is either a tautology or contingency, 
and that an unsatisfiable proposition is a contradiction. 

The DNF satisfiability problem is the problem of determining whether or not there is any 
variable assignment that makes a DNF formula true. We have the following example. 


Example 1.32 The following proposition 
(AP A Q)V (PARAR) V (=P. AAR), 


which is in DNF is satisfiable. To see this, note that the first clause evaluates to true when 
P is false and Q is true. Since the proposition is a disjunction of clauses, it is not necessary 
to evaluate the rest of the clauses. E 


Example 1.32 leads us to the following remark. 


Remark 1.5 A propositional formula in DNF is satisfiable if and only if at least one of its 
conjunctive clauses is satisfiable. A conjunctive clause is satisfiable if and only if for every 
atomic proposition P, the clause does not contain both P and =P as literals. 


From Remark 1.5, it can be seen that the DNF satisfiability is “easy.” In fact, we evaluate 
one conjunction at a time, if at least one conjunction is not a contradiction, the formula is 
satisfiable. 

The CNF satisfiability problem is the problem of determining whether or not there is 
some variable assignment that makes a DNF formula true. We have the following example. 


Example 1.33 The following proposition 
PY®QACQYRYV AS) A(AP VS), 
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which is in CNF is satisfiable. To see this, note that all clauses evaluate to true when P is 
true, Q is false, R is false, and S is true. Since the proposition is a conjunction of clauses, it 
is necessary to evaluate all clauses. a 


Ifthe number of propositional variables is very small in a CNF formula, we can determine 
whether the formula is satisfiable or not is by using a truth table. We will leave proving 
(by constructing a truth table) whether the following proposition is satisfiable or not as an 
exercise for the reader. 


(PV QOVR)A CPV 7Q) A (AP V AR) A(R V 7Q). 


In general, the CNF satisfiability problem is not as easy as this looks in Example 1.33 or 
in the above exercise. 


Definition 1.19 A propositional formula is in k-CNF if it is the AND of clauses of ORs of 
exactly k variables or their negations. 


For example, the propositional formula (P v Q) A (“œQ v AR) A (AP. v R) is in 2-CNF, and 
the propositional formula (P v Q V R) A (AQ V aR v aS) A (AP V RV S) is in 3-CNF. 

In fact, the 2-CNF satisfiability problem is a version of the satisfiability problem that can 
be formulated and solved using the so-called depth-first search method, which is the topic 
to be covered in Sections 9.4 and 9.5. However, it is worth mentioning that 3-CNF satisfi- 
ability is a “hard” problem. Further discussion about this can be found in any textbook on 
computational complexity theory. 


1.6 Predicates and Quantifiers 


In the second part of this chapter, we study the predicate logic which extends propositional 
logic by adding predicates and allowing the presence of quantifiers. The predicate logic is 
also known as first-order logic. First-order logic quantifies only variables that range over 
individuals. Second-order logic and third-order logic admit quantifications over sets and 
sets of sets, respectively. In general, higher-order logic quantifies over sets that are nested 
deeply as necessary. 


1.6.1 Predicates 


A predicate can be thought of as a function whose codomain is {T, F}. 
As a matter of examples, the following functions are predicates. 


e P(x) = “x > 4” isa predicate on one variable. Note that, for instance, P(5) is true because 
5 > 4, but P(3) is false because 3 # 4. 

e Q(x, y) = “x > y” is a predicate on two variables. Note that, for instance, Q(5,4) is true 
while Q(4, 5) is false. 


The above two examples illustrate the concept of a predicate. A formal definition looks 


like: 


Definition 1.20 A predicate is a statement whose truth value depends on the value of one 
or more variables. 


1.6 Predicates and Quantifiers 


1.6.2 Quantifiers 


Defining quantifiers leads us to define the predicate logic. 


Definition 1.21 A quantifier is an expression, such as “every” and “there exists,” that 
indicates the scope of a term to which it is attached. 


Many theorems, conjectures, and definitions in mathematics use quantifiers. For 
example, 


e “Every prime number greater than 2 is odd.” 

e “There exists an even prime number.” 

e “For any integer value of n greater than 2, there are no three positive integers a, b, and c 
that satisfy the equation a” + b” = c”.”) 


There are two fundamental kinds of quantification in the predicate logic: 


(i) The universal quantification: The quantifier that we symbolize by “Y” and read as 
“every” or “for all” is called the universal quantifier. 

(ii) The existential quantification: The quantifier that we symbolize by “3” and read as 
“there is” or “there exists” is called the existential quantifier. 


The following example helps us to always remember that the upside-down “A” stands for 
“All” and the backward “E” stands for “Exists.” Recall that N = {1, 2, 3, ...} denotes the set 
of all natural numbers. 


Example 1.34 If P(x) = “x? > 4,” then 


(a) “ax EN, P(x)” is a proposition that means “there is a natural number x such that 
x? > 4.” This is a true predicate logic proposition. 

(b) “vx € N, P(x)” is a proposition that means “for every natural number x we have x? > 4.” 
This is a false predicate logic proposition. (To see its falsity, take x = 1.) E 


In the two predicate logic propositions given in Example 1.34, the set N is called the uni- 
verse of discourse. We have the following definition. 


Definition 1.22 The universe of discourse, which is also called the domain of discourse 
(or simply the domain) and is denoted by D, is the set of the possible values of the variable(s) 
in the predicate. 


When we use quantifiers, it is important to state the universe of discourse. In Calcu- 
lus courses, unless specified otherwise, the domain of a function is usually the set of real 
numbers R. In propositional logic, the default domain is everything: All numbers, people, 
computers, cats, etc. 


11 This is called the Fermat’s last theorem (also known as Fermat’s conjecture) and was first conjectured 
by Pierre de Fermat around 1637. 
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Let P(x) be a predicate. The simplest form of a quantified formula is as follows. 


v 
{ 5 \ x ED, P(x). 
—“—— “— 
aa avariable a predicate 
a quantifier inadomain 
More specifically, 
e we write “Vx € D, P(x)” to say that a predicate, P(x), is true for all values of x in some 
set D, 
e we write “Ax € D, P(x)” to say that a predicate, P(x), is true for at least one value of x in 
some set D. 


For example, the proposition “Yn € N,n < n?,” which means that “Every natural number 
is less than its own square,” is false, while the proposition “dn € N, n = n?,” which means 
that “Some natural number is equal to its own square,” is true because n = 1 is a witness. 
Table 1.14!” shows when the truth and falsity of quantifiers are asserted. 

Some propositions have multiple quantifiers and/or multiple predicates. In this part, we 
study this concretely. 


1.6.3 Multiple Quantifiers 


More than one quantifier can be used in a proposition. Let P(x, y) be a predicate on two vari- 
ables. The simplest forms of a quantified formula with multiple quantifiers are as follows. 


Yx € D,Vy € D, P(x,y), 
ax € D, 3y € D, P(x,y). 


We have the following examples. 


Example 1.35 Let P(x,y) = “x+y = 5.” Then: 


(a) The proposition “There exist two natural numbers whose sum is 5,” or equivalently the 
more detailed proposition “There is a natural number x and a natural number y such 
that x + y = 5,” can be symbolized as Ax € N, dy € N, P(x,y). This is a true proposition. 

(b) The proposition “The sum of every pair of natural numbers is 5,” or equivalently the 
more detailed proposition “For any two natural numbers x and y we have x + y = 5,” 
can be symbolized as Yx € N, Vy € N, P(x, y). This is a false proposition. To see its falsity, 
take the pair of numbers 1 (to be x) and 2 (to be y). m 


Table 1.14 The truth and falsity of quantifiers. 


Quantifiers Universal quantifier Existential quantifier 
Formulas Vx E D,P(x) ax € D, P(x) 

When true? If P(x) is true for every x If there is an x s.t. P(x) is true 
When false? If there is an x s.t. P(x) is false If P(x) is false for every x 


12 In Table 1.14, “s.t.” refers to “such that.” 
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Example 1.36 Let P(x,y) = “x+y = 2” and Q(x, y) = “xy > 0.” Decide whether each of 
the following propositions is true or false. Justify your answer. 


(a) dx € N, 3y EN, P(x,y). (c) Vx E€ N, Yy E N, Q(x, y). 
(b) 3x € N, 3y E N, P(x,y) A x £ y. (d) Yx € Z, Vy E Z, Q(x, y). 
Solution 


The proposition in item (a) is true because x and y can both be 1. For item (b), note that “4” 
has higher precedence than “A.” The only assignment for x and y that makes the proposition 
in item (b) true is x = y = 1. Thus, the proposition in item (b) is false. Items (c) and (d) are 
left as an exercise for the reader. | 


1.6.4 Multiple Predicates 


Some propositions have more than one predicate. Let P(x) and Q(x) be two predicates, the 
simplest forms of a quantified formula with multiple predicates are as follows. 


Vx E D, P(X) ° Q(X), 
dx € D, P(x) ° QX), 


“oo 


where “o” is any binary logical operator such as A, V, ®, >, and +. For example, the propo- 
sition “Some natural number is equal to its own square and is equal to its own cube.” This 
can be symbolized as In EN, n = n? An = n°. 

Note that “=” has higher precedence than “A.” So, In E€ N,n=n? an =n? means 
dn E€ N, (n = n?) A(n = n°). In general, mathematical symbols such as addition, subtrac- 
tion, multiplication, division, equality, and inequality have higher precedence than logical 
operators such as negation, conjunction, disjunction, implication, and double implication. 

When a quantifier is used on the variable x we say that this occurrence of x is bound. 
When the occurrence of a variable is not bound by a quantifier or set to a particular value, 
the variable is said to be free. The part of a logical expression to which a quantifier is applied 
is the scope of the quantifier. So, a variable is free if it is outside the scope of all quantifiers. 
For example, if we consider the statement (Vx € D, P(x)) A Q(X), the x in P(x) is bound by 
the universal quantifier, while the x in Q(x) is free. The scope of the universal quantifier is 
only P(x). In fact, the statement (Vx € D, P(x)) A Q(x) is not a proposition since there is a 
free variable. 

The quantifiers “Y” and “3” have higher precedence than logical operators. For example, 
Vx E D, P(x) A Q(x) means (Vx € D, P(x)) A Q(x), and it does not mean Vx € D, (P(x) A Q(x)). 

Some of the questions that arise now are the following. 


e Is Vx E D, (P(X) A Q(x)) logically equivalent to Vx € D, P(x) A Vx € D, Q(x)? 
e Is Vx E D, (P(X) v Q(x)) logically equivalent to Vx € D, P(x) v Vx € D, Q(x)? 


Before answering these questions, we need the following definition. 


Definition 1.23 Two predicate logic propositions S and T are logically equivalent if they 
have the same truth value regardless of the interpretation.!* 


13 That is regardless of the meaning that is attributed to each propositional function, and regardless of the 
domain of discourse. 
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Proving that the proposition Vx € D, (P(x) A Q(x)) is logically equivalent to Vx € D, P(x) A 
Yx € D, Q(x) is left as an exercise for the reader. To disprove that Vx € D, (P(x) v Q(x)) is 
logically equivalent to Vx € D, P(x) V Vx € D, Q(x), we give the following counterexample 
to the assertion that they have the same truth value for all possible interpretations. Let D 
denote the set of people in the world, P(x) =“x is male,” and Q(x) =“x is female.” Then Vx € 
D, (P(x) v Q(x)) is true, while Vx € D, P(x) v Vx € D, Q(x) is false. In fact, the first proposi- 
tion means that every person in the world is a male or a female, while the second one means 
that every person in the world is a male or every person in the world is a female. Note that 
the formula Vx € D, P(x) V Vx € D, Q(x) can also be written as Vx € D, P(x) v Vy € D, QQ) 
or as Vx,y E D, (P(X) v Q)). 

Consider also the following two assertions. 


e Ax E D, (P(X) A Q(x)) is logically equivalent to 4x € D, P(x) A Ax E D, Q(x). 
e Ax E D, (P(X) v Q(x)) is logically equivalent to 4x € D, P(x) v Ax € D, Q(x). 


The first assertion is false, and a counterexample is D = Z, P(x)= “x <0,” and 
Q(x) =“x > 0.” Stating whether the second assertion is true or false and providing a 
proof or a counterexample are left as an exercise for the reader. 


1.6.5 Mixing Quantifiers 


Some propositions have two different kinds of quantifiers. Let P(x, y) be a predicate on two 
variables. The simplest forms of a quantified formula with mixed quantifiers are as follows. 


Yx € D, 3y € D, P(x,y), 


1.6 
dx € D, Vy E D, P(x,y). (1.6) 


When quantifiers of the same kind are used in a proposition, the order of the quantified 
variables does not matter. For example, let P(x, y) be a predicate on two variables, the fol- 
lowing two propositions have always the same meaning regardless of the predicate P(x, y). 


3x € D, 3y € D, P(x,y), 
dy € D, Ax € D, P(x,y). 


When both universal and existential quantifiers are used in the same proposition, the order 
can matter. For example, the first proposition in (1.6) and the proposition 


dy € D, Yx € D, P(x,y) 
can have different meanings. Also, the second proposition in (1.6) and the proposition 
Vy € D, Ax € D, P(x,y) 


can have different meanings. For more illustration, we give the following example. 


Example 1.37 Let P(x,y) = “x +y = 0.” Then: 


(a) The proposition Vx € R, 3y € R, P(x,y) means that for any real number x there exists a 
real number y such that x + y = 0. This is clearly a true proposition because y = —xisa 
witness. 


1.6 Predicates and Quantifiers 


(b) The proposition 3x € R, Vy € R, P(x,y) means that there is a real number x such that 
for every real number y we have x + y = 0. This is clearly a false proposition because 
there does not exist a real number whose sum with every real number is equal to 0. 

a 


Some propositions have two or more domains of discourse, one for each kind of quanti- 
fiers, as we will see in the following example. 


Example 1.38 Symbolize Goldbach’s conjecture which states that “Every even integer 
greater than 2 is the sum of two primes.” 


Solution 
Goldbach’s conjecture can be restated as “For every even integer n greater than 2, there exist 
primes p and q such that n = p + q.” 

Let Evens be the set of even integers greater than 2 and Primes be the set of prime 
numbers. Then Goldbach’s conjecture can be symbolized as 


Vn € Evens, dp € Primes, dq € Primes,n = p +q. 


This answers the question raised in the example. a 


1.6.6 Negating Quantified Statements 


In this part, we study how to negate quantified statements. 
The negation of a universal statement is an existential statement with the predicate 
negated. That is, 


(Vx, P(x)) = Ax, aP(x). (1.7) 


Often, a proposition such as =(Vx, P(x)) is simply written as “Vx, P(x). As a direct applica- 
tion, we have the following example. 


Example 1.39 The proposition “Every prime number is odd” is false because 2 is a prime 
number that is not odd. So its negation, which is the proposition “There exists a prime 
number that is not odd,” is true. 

Symbolically, to see why the second proposition is the negation of the first one, let Vx be 
“For every prime number x” and P(x) be “x is an odd number.” The negation of the propo- 
sition “Every prime number is odd” is expressed as =Vx, P(x) or equivalently using (1.7) 
dx, P(x). This statement, when subjected to a process of linguistic transformation, is 
subsequently rendered into the English language as “There exists a prime number that is 
not odd.” a 


The negation of an existential statement is a universal statement with the predicate 
negated. That is, 
7(Ax, P(x)) = Vx, aP(x). (1.8) 


Often, a proposition such as 7(4x, P(x)) is simply written as 74x, P(x). As a direct applica- 
tion, we have the following example. 


35 


36 


1 Mathematical Logic 


Example 1.40 The proposition “There is a negative integer whose square is negative” is 
clearly false. So its negation, which is the proposition “The square of every negative integer 
is Nonnegative,” is true. 

Symbolically, to see why the second proposition is the negation of the first one, let 4x 
be “There is a negative integer x” and P(x) be “The square of x is negative.” The negation 
of the proposition “Every prime number is odd” is expressed as 73x, P(x) or equivalently 
using (1.8) Vx, =P(x), which is translated back to English as “The square of every negative 
integer is Nonnegative.” E 


1.6.6.1 Negating Propositions with Multiple Quantifiers 

The process of negating quantified statements is not limited solely to those propositions 
that feature a single quantifier; it can be expanded and applied to more complex proposi- 
tions that involve multiple quantifiers. To illustrate this, consider a scenario where we have 
a predicate denoted as P(x, y, z), which operates on three distinct variables. To negate the 
proposition Vx E€ D, 3y € D, Vz E€ D, P(x, y, zZ) effectively, we employ the following sequence 
of logical equivalences. 


(Vx € D, 3y € D, Yz € D, P(x, y,z)) = Ix € D, ~= (Ay € D, Vz € D, P(x, y,2)) 
ax € D, Vy € D, ~ (Yz € D, P(x, y, 2) 
= Jx € D,Vy € D, 3z € D,7P(x,y, 2). 


Example 1.41 The proposition Vx € R, dy € R, xy = 1 means that “Every real number 
has a multiplicative inverse.” This is false because picking x = 0, then for any y € R, xy £ 1. 
To negate the proposition Yx € R, 3y € R, xy = 1, we can proceed as follows. 


a(x E€ R, 3y € R, Xy = 1) = 3x € R, WAGER, xy = 1) 
Jx € R, Yy E R, a (xy = 1) 


3x € R,Vy E R, xy #1, 


which is the desired negation. E 


1.6.6.2 Expressing Propositions Using Only One Type of Quantifiers 
On certain occasions, for the sake of simplicity and ease of expression, it becomes necessary 
to formulate propositions or statements using just a single type of quantifier, which can be 
either the universal quantifier or the existential quantifier. This approach to proposition 
construction simplifies the language and allows for more straightforward and concise com- 
munication of ideas or concepts without the added complexity that may arise from using 
both types of quantifiers within a single proposition. 

In fact, every universal statement can be expressed as an existential statement as follows. 


Vx E D, P(x) = 77Vx € D, P(x) = 73x € D, ~aP(x). (1.9) 


As a direct application, we have the following example. 


Example 1.42 The proposition “Every US senator is at least 30 years old” is the same as 
the proposition “There does not exist a US senator who is not at least 30 years old.” 
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Symbolically, to see why the first proposition is equivalent to the second one, let Vx be 
“Every US senator x” and P(x) be “x is at least 30 years old.” The proposition “Every US 
senator is at least 30 years old” is expressed as 


Vx € D, P(x) or equivalently using (1.9) 74x € D, =P(x), 
which is translated back to English as “There does not exist a US senator who is not at least 
30 years old.” a 
Similarly, every existential statement can be expressed as a universal statement as follows. 


dx € D, P(x) = 773x € D, P(x) = 7Vx E D, ~aP(x). (1.10) 


As a direct application, we have the following example. 


Example 1.43 The proposition “There exists a white cat” is the same as the proposition 
“Not every cat has color different than white.” 

Symbolically, to see why the first proposition is equivalent to the second one, let 3x be 
“There exists cat x” and P(x) be “x has the color white.” The proposition “There exists a 
white cat” is expressed as 


dx € D, P(x) or equivalently using (1.10) =~Yx € D,=P(x), 
which is rendered in English as “Not every cat has color different than white.” a 


Note that the above procedure can be extended to propositions with more than one quan- 
tifier. For example, let P(x, y, z) be a predicate on three variables, then 


Vx € D, Ay E€ D,Vz € D, P(x, y, Zz) = Vx € D,774y €E D,Vz E D, P(x, y, Z) 


= Vx € D,7Vy € D, =z E D, P(x,y, 2). 


1.7 Symbolizing Statements of the Form “All P Are Q” 


In this section, we study modeling of the statements of the form “All P are Q” based on the 
predicate logic. Perhaps the best way to understand how to symbolize this form of predicate 
logic propositions is to give illustrative examples. 

The proverb saying “All that glitters is not gold” means that not everything that is shiny 
and superficially attractive is valuable, that is, appearances can be deceiving. On the con- 
trary, Led Zeppelin in the song “Stairway to Heaven” said “All that glitters is gold.” In 
the following example, inspired by Exercise 3 in (Pugh, 2003, Chapter 1), we express the 
statement “All that glitters is gold” and its negation using quantifiers. 


Example 1.44 


(a) Symbolize the statement “All that glitters is gold.” 
(b) Symbolize the statement “All that glitters is gold” using only an existential quantifier. 


14 Led Zeppelin were an English rock band formed in London in 1968. 


38 


1 Mathematical Logic 


(c) 


Symbolize the negation of the statement “All that glitters is gold” and translate it back 
to English. 


(d) Symbolize the statement “Nothing that glitters is gold.” 


Solution 
We let D be the set of all elements, whether they are gold or not, whether they glitter or not. 
We also define the two predicates: 


(a) 


(b) 


(c) 


GLT(x) = “x glitters” and GLD(x) = “x is gold.” 


The statement “All that glitters is gold” means that “If x glitters, then x is gold.” This is 
an implication, so the correct way to express it as 


Yx € D,(GLT(x) ——> GLD). (1.11) 


By the implication law, the formulation in (1.11) is equivalent to the following 
formulation. 


Yx € D,(AGLT(x) v GLD(x)). (1.12) 


It is worth mentioning that a common error is to symbolize the statement “All that 
glitters is gold” as Vx € D, (GLT(x) A GLD(x)), which is not correct because it means 
that everything in the universe both glitters and is gold. Another error that some people 
make is to symbolize “All that glitters is gold” as Vx E€ GLT(x), GLD(x), which is not 
correct because “€” is a set operator but GLT(x) is a predicate, not a set. 

From (1.12), it is seen that the statement “All that glitters is gold” is symbolized as 
Yx € D,(AGLT(x) v GLD(x)). The following sequence of logical equivalences expresses 
this formula using only an existential quantifier. 


Yx € D,(AGLT(x) v GLD(x)) 


~~Yx € D, ((GLT@œ) v GLD(x)) 
= 74x € D,->(-GLT(x) v GLD(x)) 
= 73x € D, (GLT) A "GLD(x)) 
= 74x € D,(GLT(x) A ~GLD(x)). 
Here, the first equivalence follows from the double negation law, the second equiv- 
alence follows from (1.7), and the last two follow from DeMorgan’s law and double 
negation law. 
From (1.11), the negation of the statement “All that glitters is gold” is symbolized as 
aVx E D, (GLT) v GLD(x)). Using the rules of propositional logic, we have 
aVvx E D,(AGLT(x) v GLD(x)) = 3x € D,>(-GLT(x) v GLD(x)) 
= dx € D,(-7GLT(x) A ~GLD(x)) 
= Jx € D,(GLT(x) A nGLD()), 


which is translated back to English as “There is something in the universe that glitters 
but is not gold.” 
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(d) Using the predicates defined above, the statement “Nothing that glitters is gold” is sym- 
bolized as Vx € D,(GLT(x) > ~GLD(x)). 


This answers the questions raised in the example. a 


In the following example, we encounter a statement asserting the absence of a specific 
condition. In the first part, we are tasked with transforming this statement into the formal 
structure of “All P are Q.” The second part takes a different turn as we examine the concepts 
of logical negation and double negation. 


Example 1.45 Consider the statement “There are no orange sharks.” 


(a) Symbolize the given statement after writing it in the form “All P are Q.” 
(b) Can you obtain the same formula obtained in item (a) by symbolizing after negating 
the negation of the given statement? Justify your answer. 


Solution 
We let D be the set of all creatures (or species), whether they are sharks or not, whether 
they are orange-colored or not. We also define the two predicates: 


SRK(x) = “x isa shark” and ORG(x) = “x is orange.” 


(a) The statement “There are no orange sharks” is equivalent to the implication “If some- 
thing is a shark, then itis not orange.” So, a correct way to symbolize the given statement 
is Vx € D, (SRK(x) > AORG(x)). 

(b) We can obtain the same formula by negating the negation of the given statement. In 
fact, another way to get the same formula is to symbolize the negation of the statement 
“There is an orange shark” as follows. 

adx € D, (SRK) A ORG(x)) = Vx € D, 7SRK(x) A ORG(X)) 
= Vx € D,(7SRK(x) v 7ORG(x)) 
= Vx € D, (SRK) > ~ORG(x)) 
= Vx E D, (SRK(x) > ~ORG(x)). 
This justifies our formulation and hence answers the questions raised in the example. 
E 


Example 1.46 Symbolize the following statements using the predicate logic. 


(a) “There is at most one piano.” (c) “There are at most two pianos.” 
(b) “There are at least two pianos.” (d) “There are exactly two pianos.” 
Solution 


We let D be the set of all musical instruments, whether they are pianos or not. We also define 
the predicate: P(x) = “x is a piano.” 


(a) The statement “There are at least two pianos” is equivalent to the implication “If there 
are two musical instruments that are pianos, then they must be the same instrument.” 
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This can be symbolized as 
Vx E D,Vy E D, (P(x) A Py) —— x = y), 
or equivalently, using the implication and DeMorgan’s laws, as 
Vx E€ D,Vy € D, PŒ) v 7P(y) Vx =y). 
) An incorrect way to symbolize the statement “There are at least two pianos” is the fol- 
lowing. (Why is this incorrect?) 
Ax € D, 3y € D, (P(x) A P(y)). 
To obtain the correct formula, note that the statement “There are at least two pianos” 


is the negation of the statement “There is at most one piano.” Using item (a) and the 
rules of propositional logic, we have 


AVx E D,Vy E D, (P(x) v aP@) Vx = y) 
= dx € D,-Vy E D, (P(x) v aP) Vx = y) 
= dx € D, Ay € D,7(4P(X) v aP) Vx = y) 
= 4x E€ D, 3y E D, (P@œ@) A PO) Ax £ y). 
Thus, the statement “There are at least two pianos” is symbolized as 3x € D, 
dy € D, (P(x) A PY) Ax # y). Note that the obtained formula explicitly says that x and 
y are not the same instrument. In the earlier incorrect formula, x and y could refer to 


the same instrument. 
The statement “There are at most two pianos” is symbolized as 


Yx € D, Yy E D, Yz € D, (Pœ A PY) A PRY — (x =y) V(X =2) V(y = 2). 


(d) The statement “There are exactly two pianos” means that “There are at least two pianos 


and there are at most two pianos.” From items (b) and (c), this can be symbolized as 
[ax € D, Ay E D, (PX) A PO) Ax Ey) A 
[vx € D, Yy E D, Yz € D, PŒ) A PO) A PZ) > x=y vx =z)yv Q =z). 


An alternative way to symbolize the statement “There are exactly two pianos” is as 
follows. 


ax € D, Ay E D, (œ Fy) A [Yz E D, PQR) — [Iœ = z2) Vv 0 = 23D). m 


Finally, it is worth mentioning that sometimes statements of the form “All P are Q” 


are expressed using quantifiers and predicates with more than one variable. For example, 


to 


symbolize the statement “Every mail message larger than two megabytes will be 


compressed,” we define the predicates ML(x,k) = “The mail message x is larger than 


k 


megabyte” and MC(x) = “The mail message x will be compressed.” Then the given 


statement is symbolized as follows. 


Yx € D, (ML(x,2) ——> MC(x)). 


Here, the domain of discourse D is the set of all mail messages. 
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Exercises 


1.1 Choose the correct answer for each of the following multiple-choice question- 
s/items. 


(a) 


(b) 


(c) 


(d) 


(e) 


(f) 


(g) 


True or False. The statement “x + 2 = 7 if x = 2” is a proposition. 
(i) True. (ii) False. 
Let P be the sentence “Q is false,” Q be the sentence “R is false,” and R be the 
sentence “P is false.” Then P is 
(i) a proposition. 
(ii) not a proposition, but a paradox. 
(iii) not a proposition, but an anti-paradox. 
(iv) not a proposition, neither a paradox nor an anti-paradox. 
Let P represent a true statement, and let Q and R represent false statements. The 
truth value of the compound statement =(7P A ~Q) v (AR v ~P) is 
(i) True. (ii) False. 
Let P be the statement “Kids are happy” and Q be the statement “Parents are 
happy.” If we translate the compound proposition =(P vV =Q) into words, we get 
(i) It is not the case that kids are happy or parents are not happy. 
(ii) Kids are not happy and parents are not happy. 
(iii) It is not the case that kids are happy and parents are not happy. 
(iv) Kids are not happy or parents are not happy. 
True or False. If 6 < 1, then 11 < 4. 
(i) True. (ii) False. 
A sufficient condition that a triangle T be a right triangle is that its three sides 
satisfy a Pythagorean triple. An equivalent statement is 
(i) If T is a right triangle then its three sides satisfy a Pythagorean triple. 
(ii) If the three sides of a triangle T satisfy a Pythagorean triple, then T is a right 
triangle. 
(iii) If the three sides of a triangle T do not satisfy a Pythagorean triple, then T 
is not a right triangle. 
(iv) T is a right triangle only if its three sides satisfy a Pythagorean triple. 
Consider the following argument: 
If I am thirsty, then I will drink a glass of water. 
Iam not thirsty. 
Iwill not drink a glass of water. 
Then this argument is 
(i) valid by the implication. 
(ii) invalid by the fallacy of the inverse. 
(iii) valid by the contrapositive. 
(iv) invalid by the fallacy of the converse. 
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(h) The total number of rows in the truth table for the compound proposition 
(PV Q)A(RV AS) > T is 
(i) 25. (ii) 15. (iii) 23. (iv) 32. 
(i) Consider the statement, “If n is divisible by 12, then n is divisible by 2 and by 3 
and by 4.” This statement is equivalent to the statement 
(i) Ifnis not divisible by 12, then n is divisible by 2 or divisible by 3 or divisible 
by 4. 
Gi) Ifn is not divisible by 12, then n is not divisible by 2 or not divisible by 3 or 
not divisible by 4. 
(iii) Ifn is divisible by 2 and divisible by 3 and divisible by 4, then n is divisible 
by 12. 
(iv) If n is not divisible by 2 or not divisible by 3 or not divisible by 4, then n is 
not divisible by 12. 
Consider the statement “Given that people who are in need of refuge and conso- 
lation are apt to do odd things, it is clear that people who are apt to do odd things 
are in need of refuge and consolation.” This statement, of the form (P > Q) > 
(Q > P), is logically equivalent to 
(i) People who are in need of refuge and consolation are not apt to do odd 
things. 
(ii) People are apt to do odd things if and only if they are in need of refuge and 
consolation. 
(iii) People who are apt to do odd things are in need of refuge and consolation. 
(iv) People who are in need of refuge and consolation are apt to do odd things. 
(k) Let M = “Adam is a Math major,” P = “Adam is a Physics major,” A = “Adam’s 
wife is an Astronomy major,” K = “Adam’s wife has read Al-Khwarizmi’s alge- 
bra,” and H = “Adam’s wife has read Al-Haytham’s optics.” One of the follow- 
ing propositions expresses the statement “Adam is a Physics major and a Math 
major, but his wife is an Astronomy major who has not read both Al-Haytham’s 
optics and Al-Khwarizmi’s algebra,” which is 


G 


Ym 


G) PAM A(AV (7K V7H)). Gii) PAM AAA (7K v =H). 
(ii) PAM AAA (7K A-H). (iv) PAM A(AV (7K ^ >H)). 
(1) The negation of the proposition “If3 is positive, then —3 is negative” is the propo- 
sition 


(i) “If3 is positive, then —3 is also positive.” 
Gi) “If 3 is negative, then —3 is positive.” 
(iii) “3 is positive and —3 is positive.” 
(iv) “3 is positive and —3 is nonnegative.” 


(m) One of the following is not logically equivalent to (P A Q) V (AP A Q) v (PA 7Q), 
which is 
(i) APA 7Q. (iii) QV (PA 7Q). 


(ii) PV Q. (iv) PV (QA 3P). 


1.2 


1.3 
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(n) The contradiction law is an example of 
(i) a tautology. (iii) a contingency. 
(ii) a contradiction. (iv) none of the above. 
(0) One of the following compound propositions is in both DNF and CNF, which is 
(i) (PV QV ATR. (iii) PAQ ATR. 
(ii) (PAQ) VaR. (iv) None of the above. 
(p) True or False. A propositional formula in CNF is satisfiable if and only if at least 
one ofits disjunctive clauses is satisfiable. 
(i) True. (ii) False. 


Which of the following sentences are propositions? What are the truth values of 
those that are propositions? (Here, in items (c)-(f), x and y are any real numbers 
and 8 is any angle in the standard position.) 


(a) 24+3=5. (i) Do not be in close contact with a sick 

(b) 24+ 3. person with COVID-19. 

(c) x+2=11. (j) Repeat your answer to item (1). 

(d) x+y=y4+x. (k) Your answer to item (l) is incorrect. 

(e) cos?0 = 1. (1) The truth value of the statement in 

(£) cos?@ + sin*@ = 1. item (h) is true. 

(g) Which of the following sentences are (m) The truth value of the statement in 
propositions? item (i) is true. 

(h) John F. Kennedy is the 35th presi- (n) The truth value of the statement in 
dent of the United States. this item is false. 


What is the negation of each of the following propositions? 

(a) Today is Thursday. 

(b) There is no pollution in New Jersey. 

(c) 24+1=3. 

(d) Sara’s first answer to item (l) in Exercise 1.1 was incorrect. 

(e) The summer in Santiago is not hot. 

(f) The summer in Santiago is hot but bearable. 

(g) The summer in Santiago is not hot or it is not humid. 

(h) If the sun is shining in Santiago’s sky, then I will go to the nearest beach. 
(i) If the sun is shining in Santiago’s sky, then I will go to the nearest beach and do 

a little physical exercise. 


1.4 Let Pand Q be propositions 


P : I bought a lottery ticket this week. 
Q : Iwon the million-dollar jackpot on Friday. 
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1.5 


1.6 


1.7 


1.8 


1.9 


Express each of the following propositions as an English sentence. 
(a) ~P. (b) PVQ. (c) PAQ. (d) =P A7Q. 


Let P and Q denote the statements “It is below freezing” and “It is snowing,” respec- 
tively. Formulate the following statements using P and Q and logical connectives. 
(a) It is below freezing and snowing. 

(b) It is below freezing but not snowing. 

(c) It is not below freezing and it is not snowing. 

(d) It is either snowing or below freezing (or both). 


Consider the implication: “A passing score on the final exam is required in order to 
receive a passing grade for the course.” Restate this statement as five implications, 
all have the same meaning, and each using one of the five equivalent forms listed in 
Remark 1.3. 


The following implications are mainly about the twin primes conjecture.!> Decide 

whether each of these implications is true or false. Justify your answer. 

(a) If the twin primes conjecture is unsolved, then the twin primes conjecture is 
conjectured by a monkey. 

(b) If the twin primes conjecture is true, then Pythagoras’ theorem is true. 

(c) If the twin primes conjecture is false, then Pythagoras’ theorem is true. 

(d) If the twin primes conjecture is either proven or disproven, then the twin primes 
conjecture is solved. 

(e) If the twin primes conjecture is true, then the twin primes conjecture is true. 

(f) If the twin primes conjecture is either true or false, then the twin primes conjec- 
ture is solved. 

(g) If the twin primes conjecture is either true or false, then the twin primes conjec- 
ture is unsolved. 


Prove the result of the following theorem. Start by restating the theorem in IF-THEN 
form. 


Theorem I: The sum of two odd integers is even. 


Write the contrapositive, converse, and inverse of the proposition: “If the Sun is 
shrunk to the size of your head, then the Earth will be the size of the pupil of your 


” 


eye. 


15 Twin primes are primes that are two steps apart from each other on the number line, such as 3 and 5, 
5 and 7, 29 and 31, and so on. The twin primes conjecture (also known as the Polignac’s conjecture) states 
that there are infinitely many twin primes. This conjecture is still a mysterious unsolved problem in the 


field of Number Theory. 


1.10 


Exercises 


Give, if possible, an example of a true conditional statement for which 


(a) the converse is true. (b) the contrapositive is false. 


Let x be a positive integer. Prove the results of the following two theorems. Use the 
low of contrapositive. 


Theorem I: If x? is even, then x is even. 
Theorem II: x? is odd iff x is odd. 


Negate and simplify the following two compound propositions. 

(a) P © Q. (Hint: Use the logical equivalence P + Q = (P > Q) ^ (Q > P)). 
(b) P@ Q. (Hint: Use the logical equivalence P 6 Q = (PV Q) A ~P A Q)). 
Use truth tables to prove that Po Q = (P > Q) A (Q > P). 


Construct truth tables for the following compound propositions. 


(a) PAP. (k) P > Q)V GP > Q). 
(b) (PV =Q) > Q. () Pe QVAP e Q). 
(c) PVQ) > (PAQ). (m) (PA Q)VR. 

(d) (P > Q) e CQ > 2P). (n) (PAQAR. 

(e) POP. (0) (PV Q) VR. 

(f) PRQ. (p) (PAQ) V-R. 

(g) =P ~Q. (q) P> QVR). 

h) (P aQ) PRQ). (t) (P > Q) V (=P >R). 
(i) P >Q. (s) P > QAP = R). 
G) PeR. (t) (PS Q)V(AQ eR). 


Consider the following propositional logic word problem: Either Lillian is forceful 
or she is creative. If Lillian is forceful, then she will be a good executive. It is not 
possible that Lillian is both efficient and creative. If she is not efficient, then either 
she is forceful or she will be a good executive. Can you conclude that Lillian will be a 
good executive? Your reasoning and conclusion should be justified by linking them 
to the propositional logic. 


List the names of the logical equivalence laws from Table 1.8 that are used to prove 
Item (c) of Example 1.27. 


Determine if each of the following implications is a tautology, contradiction, or con- 
tingency. Justify your answer without using truth tables. 

(a) [PAP VQ] > Q. 

(b) [P > QA(Q>R] > P >R). 

(c) IP AP > Q] > Q. 

(d) (PV Q) A (P > R) A (Q >R) >R. 
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1.18 


1.23 


Without using truth tables, prove that the following propositions are logically 
equivalent. 


(a) PQ. (b) (PAQ VOP). (© PEQ). 


Hint: You can show that the propositions (a), (b), and (c) are logically equivalent by 
showing that the propositions in (a) and (b) are logically equivalent and that those 
in (b) and (c) are logically equivalent. 


Construct a DNF having the following truth table. 


P Q R Statement 
T T T F 
T T F T 
T F T T 
T F F F 
F T T F 
F T F T 
F F T F 
F F F F 


For each of the following statements, find an equivalent statement in CNF. 
(a) 7(A V B). (b) -(A A B). (c) Av (BACO). 


Give an example of a proposition in DNF with three variables and four distinct 
clauses that is unsatisfiable. 


Is the propositional formula P A Q A (ŒR V S V ^T) satisfiable? Explain. 


Choose the correct answer for each of the following multiple-choice question- 
s/items. 
(a) The propositions in items (c) and (d) of Example 1.36 are 
(i) both true. (iii) true and false, respectively. 
(ii) both false. (iv) false and true, respectively. 
(b) Expressing the proposition 4x € D, Vy € D,Vz € D, P(x, y, z) using only the exis- 
tential quantifier, we get 
(i) ax € D, 73y € D, 74z € D, AP(x, y, Z). 
(ii) 3x € D, 73y € D, 737 € D, P(x, y, Z). 
(iii) 4x € D, Ay € D, 75z € D, aP(x, y, Z). 
(iv) 3x € D, ~3y € D, Az € D, AP(x, y, 2). 
(c) Let V be the set of graph vertices and C be the set of all colors (see Sections 4.1 
and 4.4). Let also edge(u, v) mean that “vertex v is adjacent to vertex u,” and 
color(v,x) mean that “vertex v has color x.” Which one of the following is the 


Exercises 


correct formulation for the statement “Adjacent vertices do not have the same 
color”? 
(i) Vv E V, Vu E V, Yx E C, (edge(v, u) A color(v, x) A ~acolor(u, x)). 
(ii) Vv € V, Vu € V, Yx E C, (edge(v, u) A color(v,x) > ~color(u, x)). 
Gii) Vo € V, Vu € V, Vx E C, (edge(v, u) A color(v, x) V =color(u, x)). 
(iv) Wv € V, Yu € V,Yx €E C, (edge(v, u) A color(v, x) V color(u, x)). 


(d) Let F be the set of all (direct or indirect) flights, and define the predicate 


w 


DF(u, x, y) = “There is a direct flight u from a city x to a city y.” 


One of the following statements is a correct translation for the quantified state- 
ment ~3u € F, DF(u, Columbus, Rochester), which is 
(i) There is no flight between Columbus and Rochester. 
(ii) There is no flight from Columbus to Rochester. 
(iii) There is no direct flight between Columbus and Rochester. 
(iv) All flights from Columbus to Rochester are indirect. 
The negation of “Nothing that glitters is gold” is 
(i) “All that glitters is gold.” 
(ii) “There is something that glitters that is gold.” 
(iii) “It is not all that glitters that is gold.” 
(iv) “All that glitters that is not gold.” 
Let S be the set of all shapes, and define the three predicates F(x) = “x is a foot- 
print shape,” C(x) = “x is circular-shaped,” and E(x) = “x is elliptical-shaped.” 
One of the following is the correct formulation for the statement “Each footprint 
shape is either circular or elliptical, but not both,” which is 
(i) Vx E S, (F(x) — [Cœ v E(x))). 
(ii) Yx € S, (F(x) — [C(x) > RE]. 
(iii) Yx € S, Fœ — [C(x) e AE(x))). 
(iv) Vx € S, (Fx) —> [C@) ® 7E(x))). 


(e 


Ym 


(f 


Ym 


1.24 Let the domain of discourse consist of the nonzero integers, that is, D = Z — {0}. 


Give the truth value of each of the following quantified statements. Find a value of 
x that supports your conclusion. 


(a) Vx € D,x < 2x. (d) Wer Č <x. 
(b) Sx € D,x+x=x-—x. 5 
(c) Ax € D, x? = 2x. (e) ax € D, = =x. 


Consider the following two propositions. 
dx € N, Vy E N, P(x,y), (1.13) 
vx € N, ay EN, P(x,y). (1.14) 


Using the above two propositions, give a predicate P(x, y) that makes (or explain why 
it is not possible): 

(a) Propositions (1.13) and (1.14) are both true. 

(b) Propositions (1.13) and (1.14) are both false. 
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(c) Proposition (1.13) is true and Proposition (1.14) is false. 
(d) Proposition (1.13) is false and Proposition (1.14) is true. 


1.26 Negate the false proposition 3x € R, Vy E R,x +y = 0. 


1.27  Symbolize the following statements using the predicate logic. Use the four predi- 
cates A(x) = “x is an architect,” C(x) = “x is a constructor,” I(x) = “x is invited,” and 
L(x) = “x is late.” 
(a) There is at least a person who is on time. 
(b) There is at least an invited person who is neither a constructor nor an architect. 
(c) All architects and constructors invited to the party are late. 


1.28 Let D be the universe of all living things, and define the three predicates: 
Child(x) = “x is a child,” Mime(y) = “y is a mime,” and Like(x,y) = “x likes y.” 
Symbolize the negation of the quantified statement “Some children do not like 
mimes.” 


1.29 Symbolize the following statements using the predicate logic. 
(a) For each owner, there is at least a house with this owner. 
(b) Adjacent houses do not have the same owner. 
(c) Each house has at most one owner. 


1.30 Symbolize the following statements using the predicate logic. 
(a) Friends of friends are friends. 
(b) Friendless people smoke. 
(c) Smoking causes cancer. 
(d) If two people are friends, either both smoke or neither does. 


Notes and Sources 


Mathematical logic has a rich history dating back to ancient civilizations. The origins of 
mathematical logic can be traced to the work of ancient Greek philosophers like Aris- 
totle in the 4th century BCE. However, the modern formalization of mathematical logic 
began in the 19th and early 20th centuries with the groundbreaking contributions of math- 
ematicians and logicians such as George Boole, Augustus De Morgan, and Gottlob Frege. 
Boole’s book Boole (1854) laid the groundwork for Boolean algebra, a crucial component of 
mathematical logic. Additionally, Frege’s book Frege and Angelelli (1977) provided a rig- 
orous symbolic language for logical propositions and introduced the concept of predicate 
logic. These early contributions set the stage for the further advancement of mathematical 
logic in the 20th century. 

This chapter offered a comprehensive introduction to the fundamental branches of math- 
ematical logic, which served as the basis for reasoning in mathematics, computer science, 
and philosophy. More precisely, it provided a valuable resource for anyone who sought to 
gain a firm grasp of propositional and predicate logic, from their foundational concepts to 
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practical applications. Before transitioning to the topic of bridging propositional logic with 
predicate logic, a shift occurred in the chapter’s focus to address a central problem within 
the realm of computer science, namely the satisfiability problem. 

As we conclude this chapter, it is worth noting that the cited references and others, such 
as Cusack and Santos (2021); Pinter (2014); Rosen (2002); Kunen (1983); Fraenkel et al. 
(1973); Scheurer (1994); Jech (1997); Schwartz et al. (2011); Krajicek (1995); Katsumata 
(2005); Moss (1981); Tiitiincii et al. (2003); Clark (1980); Al-Yahya et al. (2023); Arvind 
and Guruswami (2022); Yin et al. (2021); Dovier (2019), and Knuth (1997), also serve as 
valuable sources of information pertaining to the subject matter covered in this chapter. 
Exercises 1.14, 1.17, and 1.18 are due to Rosen (2002). Exercise 1.30 is due to Domingos and 
Lowd (2009). 
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Set theory concepts that we study in this chapter provide the fundamental underpinning 
for a wide range of theoretical structures within the fields of mathematics and computer 
science. The chapter begins with mathematical induction, which is widely used in establish- 
ing the validity of statements, laying the groundwork for sound mathematical reasoning. 


2.1 Induction 


Mathematical induction is a powerful and elegant technique for proving certain types of 
mathematical statements. As explained in Figure 2.1, there are two steps involved in knock- 
ing over a row of dominoes. These steps are the same as the steps in a proof by induction. 
See Figure 2.1 which shows the idea of induction. 

We have an infinite number of claims that we wish to prove: Claim (1), Claim (2), 
Claim (3), ... , Claim (n), .... We have the following workflow principle. 


Workflow 2.1 If we can perform the following two steps, then we are assured that all 
these claims are true: 


(i) Prove that Claim (1) is true. 
(ii) Prove that, for every natural number k, if Claim (k) is true, then Claim (k + 1) is true. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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In Workflow 2.1, Step (i) represents the knocking over the 
first domino and Step (ii) shows that if the kth domino is 
knocked over, then it follows that the (k + 1)st domino will be 
knocked over. 


The process of toppling a sequence of dominoes entails a 
twofold operation: Initially, the first domino must be tipped 
over, and secondly, it is essential to ensure that when domino 
k succumbs to gravity and topples, it effectively instigates 
the fall of domino k +1. The steps involved in this domino 
scenario notably parallel the fundamental steps of a proof 
by induction, where an initial base case is established, and a 
recursive progression is established to demonstrate that a given 
statement holds true for all values within a defined sequence. 
This intriguing analogy draws a striking resemblance between 
these seemingly unrelated actions, highlighting the unifying 
principles of causation and sequential causality. 

To apply the principle of mathematical induction to sets of Figure 2.1 Falling 
natural numbers, let S be a set of natural numbers (in other dominoes. 
words, S is a subset of N) such that 


(ij) 1 € S, and (ii) foreachk E N,kE€S—>k+1E€s, 


then S = N. The intuitive justification is as follows: By (i), we know that 1 € S. Now apply 
(ii) with k = 1, we have 2 = 1 + 1 € S. Now, apply (ii) with k = 2, we get 3 = 2 + 1 € S. And 
so forth. 


2.1.1 Principle of Induction for Predicates 


We have the following principle. 


Principle 2.1 Let P(n) be a predicate whose domain is N such that 
(i) PCL) is true, and 
(ii) if P(k) is true, then P(k + 1) is true, for all k € N. 


If (i) and (ii) are true, then P(n) is true for all n € N. 


In Principle 2.1, we call (i) the base case (basis step) and (ii) the inductive step, which 
consists the inductive hypothesis and the inductive conclusion. 


Example 2.1 Use mathematical induction to prove that the sum of the first n positive 
integers is n(n + 1)/2. 


Solution 
By induction, let P(n) be 


n(n + 1)” 
2 


“1424+3+---¢n= (2.1) 


2.1 Induction 


for n € N. First, we look at the base case for n = 1 to see whether or not P(1) is true. Substi- 
tuting n = 1 for both sides of (2.1), we get 1 = 1(1 + 1)/2 = 1. 

Next, we look at the inductive step for n = k to see whether or not P(k + 1) is true if the 
inductive hypothesis P(k) 


“LH2434etk= ESD (2.2) 
is true. Using the inductive hypothesis (2.2), we have 
k(k+1 
14+2+3+---+k+(k+1) = os 441 
= kk+1) 2+ 1) 
~~ 3 2 
— kR+1)+2K4+1)  (kK+1)(k +2) 
2 2 


Thus, if P(k) is true, then P(k + 1) is true for any k greater than or equal to 1 which proves 
the inductive conclusion. 

Therefore, by induction, for each n € N, P(n) is true. In other words, the equality in (2.1) 
holds. The proof is complete. a 


Corollary 2.1 As one of the most known series, we have! 


n 
$k = İnn +1, 
k=1 2 
which is well known in mathematics as the arithmetic series. 


Example 2.2 Use mathematical induction to prove that the sum of the first n positive odd 
integers is n?. 


Solution 
By induction on n, let P(n) be 


“T43454---+(2n-1) =n” (2.3) 


for n € N. First, we look at the base case for n = 1 to see whether or not P(1) is true. Substi- 
tuting n = 1 for both sides of (2.3), we get 2n — 1 = 2(1)- 1 = 1. 

Now, we look at the inductive step for n = k to see whether or not P(k + 1) is true if the 
inductive hypothesis P(k) 


“T43454---4+(2k—-1) =k?” (2.4) 
is true. Using the inductive hypothesis (2.4), we have 


143454---4+(2k-1)+ (k+1)-1) = k +(2(k+1)-1) 
= k?+(2k4+1)=(k +1). 


1 The symbol > is the capital Greek letter “sigma.” We write ));_, k (read “the sum of k from k equals 1 to 
k equals n”) to indicate the sum 1+2+---+n. 
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Thus, if P(k) is true, then P(k + 1) is true for any k greater than or equal to 1 which proves 
the inductive conclusion. 

Therefore, by induction, for each n € N, P(n) is true. In other words, the equality in (2.3) 
holds. This completes the proof. E 


2.1.2 Induction Proves Recursion 


A recurrence is a well-defined mathematical function written in terms of itself. 
For example, it can be shown (see Exercise 2.5) that the following recurrence 


T(1)=1, T(n) = 3T(n-1)4+4, n=2,3,..., (2.5) 
describes the function 
T(n) = 3"-2, n=1,2,.... (2.6) 


Recurrences will be studied extensively in Chapter 5. In this part, we learn briefly how to 
use induction to verify solutions of recurrence formulas. 

The formula (2.6) is called the solution of the recurrence relation (2.5) and can be 
obtained by using the method of mathematical induction. A simpler example is the 
following. 


Example 2.3 Use the induction method to show that the recurrence 


1, ifn = 0; 
T= | ayaa ifn =1,2,3,..., oe 


has the solution 
T(n)=n+1, n=0,1,2,.... 


Solution 
By induction on n. First, the base case for n = 0 is seen to be true as T(0) = 1 = 0 + 1, where 
the first equality follows from (2.7). Now, assume that the inductive hypothesis holds for 
n = k, that is, T(k) = k + 1. Then 

T(k+1)=T(k)+1=(k+1)+1=k+2. 


Therefore, if T(k) = k + 1, then T(k + 1) = k + 2 for any integer k > 0. This proves the 
inductive conclusion. Thus, by induction on n, the equality in (2.7) holds. E 


2.2 Sets 
Much of mathematical discrete structures are written in terms of sets. The concept of a 
set is used throughout mathematics and its formal definition closely matches our intuitive 


understanding of the word. 


Definition 2.1 A setis an unordered collection of distinct objects. 


2.2 Sets 


We can build sets containing any objects that we like, but usually, we consider sets whose 
elements have some property in common. 


Definition 2.2 The objects comprising a set are called its elements or members. 


Note that sets do not contain duplicates and that the order of elements in a set is not 
significant. A finite set is a set that has a finite number of elements. An infinite set is a set 
with an infinite number of elements. By convention, a set can be defined by enumerating 
its components in curly brackets. We have the following example. 


Example 2.4 The following are examples of sets. 


e Vowels = {a, e, i, 0, u}. 

e Weekend = {Saturday, Sunday}. 
e Days = {1,2,3,...,365}. 

e N= {1,2,3,...}. 

e Z = {... ,—3, —2, —1,0, 1, 2,3, ...}. 
e Truth-Values = {true, false}. 


Note that the above sets are finite, except N and Z which are infinite sets. E 


2.2.1 Set Membership 


To express the fact that an object x is a member of a set A, we write x € A. Note that an 
expression with this form is either true or false; the thing is either a member or not. For 
instance, in Example 2.4, the statements u € Vowels, 10 € N,21 € Days are true, but the 
statement —21 € Days is false. 

Note that an expression such as {1,2,3} € {4,5,6} is not allowed, but expressions such 
as {1,2,3} € {{6, 7, 8}, {1,2,3}} and {1,2,3} € {{6,2,1}} are allowed. (Why? And which 
one is true?) 


2.2.2 Cardinality of Sets 


The cardinality of a set A, denoted by |A|, is the number of elements in it. For instance, in 
Example 2.4, |Vowels| = 4 and |Days| = 365. A finite set is a set that has a finite number 
of elements. Therefore, in Example 2.4, Vowels and Days are finite sets, but N and Z are 
infinite sets. 


2.2.3 Set Equality 


Two arbitrary sets A and B are equal, written as A = B, if A and B contain precisely the same 
members. We emphasize that the order in which values occur in the set is not significant. 
For instance, in Example 2.4, the equality {1, 2,3} = {2,3,1} is true, but the equality Days 
= N is false. 
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2.2.4 Subsets and Proper Subsets 


Let A and B be arbitrary sets of the same type. Then A is said to be a subset of B, 
denoted by A CB, if every member of A is also a member of B. For example, the 
expressions {1,2,3} CN, NC Z and {1,2,3} C {1,2,3} are true, but the expressions 
{1,2,3,4} C {1,2,3} and Z CN are false. Note that equal sets are subsets of each other. 
That is, A = B © (A C B) A (B C A). 

A set A is said to be a proper subset of a set B, denoted by A C B, if A C B and A # B. For 
instance, in Example 2.4, the expression {Saturday} C Weekend is true, but the expression 
{Monday} c Weekend is false. 


2.2.5 The Empty Set 


The empty set, denoted by { } or Ø, is a special set that has the property of having no mem- 
bers. Note that nothing is a member of the empty set. That is, if x is any element, then the 
expression x € Ø must be false. Therefore |Ø] = 0, while |{@}| = 1 for instance. Note also 
that the empty set is a subset of every set. That is, if A is an arbitrary set, then Ø C A. This 
includes of course Ø C Ø. 


2.2.6 The Powerset Operator 


Let A be an arbitrary set. The set of all subsets of A is called the powerset of A and is denoted 
as P(A). So, B € P(A) — BCA. 

For example, if A = {1},B= {1,2}, and C = {a,b,c}, then P(A) = {@, {1}}, P(B) = 
{O, {1}, {2}, {1,2}}, and P(C) = {G, {a}, {b}, {c}, {a,b}, {a,c}, {b,c}, {a, b,c}}. Note that 
P(A) = 2,|P(B)| = 4 = 2? and |P(C)| = 8 = 2°. In general, for any finite set A, one can 
prove that |P(A)| = 241, Proving this is left as an exercise for the reader (see Exercise 2.4). 
See also Example 6.5. 


2.2.7 Manipulating Sets 


Venn diagrams employ intersecting circles or other geometric forms to visualize the logical 
relationships among two or more sets. See Figure 2.2 which shows Venn diagrams for the 
most common combined sets obtained from two sets A and B, such as the union of A and 
B, denoted by A U B, the intersection of A and B, denoted by A N B, the complement of A, 
denoted by A’, the difference of A and B, denoted by A — B, and the symmetric difference 
of A and B, denoted by AA B. 


Table 2.1 shows a list of most common laws of algebra of sets. 
For example, 


e {a,e,i} U {o,u} = {a,e,i,0, U}, e NnNZEN, 

e {a,e} U {e,i} = {a,e, i}, eNUZ=Z, 

e OU {a,e} = {a,e}, e {a,e,i,0,u} — {0} = {a,e,i,u}, 
e {a,e,i} N {o,u} = Ø, e Ø -— {a,e,i,u} = Ø. 


The Cartesian product of two sets A and B, denoted A x B, is the set of all possible ordered 
pairs where the elements of A are first and the elements of B are second. For example, 
{a, b,c} x {1,2} = {(a, 1), (a, 2), (b, 1), (b, 2), (c, 1), (c, 2)}. 


2.2 Sets 


AUB AAB 
A B A B 
A A-B=AQB 
AAB 
) | 7 


A AB =(AUB) = (A AB) 


Figure 2.2 Venn diagrams for combined sets obtained from two sets A and B. 


2.2.8 Sets Defined by Predicates 


The set of all elements x in a domain D such that a predicate P(x) is true is denoted as 
{x € D : P(x)} and is called the set-builder notation. For example, if P(x) is 


“x is an even integer between 1 and 11” 
then the set {x€ Z : P(x)} = {2,4,6,8,10}. For another example, the set {n EN : 
(n > 1) A (n < 4)} is the set {1, 2, 3} in the set-builder notation. We also have the following 
example. 


Example 2.5 Redefine each of the following sets using the set-builder notation. 


(a) A= {-1,z}. (b) B = (-3, 5]. (c) Ø. (d) R. 
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Table 2.1 A list of the most common laws of algebra of sets. Here, A, B, and 
C are finite sets in a space S. 


Name Formula(s) 
Identity laws AUA=A 
ANA=A 
Domination laws ANG=9 
AUS=S 
Idempotent laws ANA=A 
AUA=A 
Tautology law AUA'=S 
Contradiction law ANA’ =Ø 
Double negation law A=(A’)' 
DeMorgan’s laws (AUB! =A'NB’ 


(ANB =A'UB' 
A-—(BUC)=(A-B)n(A-C) 
A-(BNC)=(A-B)U(A-C) 


Contrapositive law (A C B)=(B' CA’) 

Commutative laws AUB=BUA 
ANB=BNA 

Associative laws (AUB)UC=AU(BUC) 


(ANB)NC=AN(BNC) 


Distributive laws (AUB)NC=(ANB)U(ANC) 
(AN B)UC=(AUB)N(AUC) 
More laws (A-B)NB=9 
(A-B)UB=AUB 
A-(BNC) =(A-B)U(A-C) 
A-(BUC) =(A-B)nN(4-C) 
An(B—-C) =(AnB)-(AnG) 
An(BAC) =(AnNB)A(ANC) 
(A — B)U(B—A) = (AUB) — (A N B) 


Solution 
It is easy to see that the given sets can be redefined as follows. 


(a) A={xER: (X+)(X-2z) =O}. (c) @={xEZ:x+x=0}. 
(b) B={xER: (-3< x) A (x < 5)}. (d) R={xER:x?4+1>0}. 


The desired answer is obtained. 


2.3 Relations 


Let S be a space that contains the sets A and B. We can write the union, intersection, 
complement, difference, and Cartesian product introduced above in set-builder notation as 
follows. 


AUB= {xES:xEA V xEB}, 


ANB= {xES:xE€A A xeEB}, 

A’ ={xES:x€A}, 
A-B={xES:xEAAXEB}, 
AAB= {xES:xE€A-Bv xEB-A}, 
AxB = {(a,b):aec AAbDEB}. 


2.3 Relations 


The concept of a relation is important in mathematics and computer science. For example, 
relations are used in the definition of functions which will be introduced in Section 2.5. 
Relations can be also used in the definition of directed graphs which will be introduced in 
Section 4.5. 

Given a set of objects, we may want to say that certain pairs of objects are related in 
some way. For example, we may say that two students are related if they are attending the 
same university or working for your company, or if they are from the same hometown. 
Mathematically speaking, we have the following definition. 


Definition 2.3 A binary relation & on two sets A and B is a subset of the Cartesian prod- 
uctA x B. The notation a & bis read a is &-related (or simply related) to b and it means that 
(a,b) E &. If (a,b) € Z, we writea F b. A relation from A to A is called a relation on A. 


There are several ways to representa relation in a usable form. For example, we can list the 
ordered pairs inside set brackets, construct its corresponding table, or plot its corresponding 
rectangular coordinate graph. We can also describe a relation with an expression such as 
an equality or inequality. We have the following example. 


Example 2.6 Let A= {0,1,2,3,4} and 
B= {0,1,2,3}. We represent a binary relation 
& from the set A to the set B in three different 
ways. We first describe & by explicitly listing its 
ordered pairs inside set brackets: 


lex] 


& = {(1, 1), (2, 0), (3, 3), (4, 2)}. 3 e 


In table form, & can be represented by the table 
shown to the right. In this tabular representa- 
tion, each ordered pair is illustrated, providing 
a structured and organized means of visualiz- 
ing the relation &. In graphical representation, 
& can be represented by the graph in the figure 1 3 

shown to the right. E 


59 


60 


2 Set-Theoretic Structures 


In general, there are many different relations from a set A to a set B, because every subset 
of A x Bis a relation from A to B. This includes the empty set Ø, which is called the empty 
relation from A to B, and includes the set A x B, which is called the full relation (or universal 
relation) from A to B. 

Definition 2.4 The domain of the binary relation & from A to B is the set 
Dom(&) = {x € A: there exists y E€ B such that xZ y}. 
The range of the relation & is the set 


Rng(Z) = {y EB : there exists x € A such that x y}. 


Example 2.7 The graphs of the following binary relations are shown below. 


R= {Hx ERXRi xp +x, <1}, FJ ={&,x%)ERXR: max {|x|, l} <1}. 


Dom(R) = Rng(R) = [-1, 1]. Dom(J) = Rng(J) = [-1, 1]. 
E 


Definition 2.5 An n-ary relation on sets A,,A,,...,A, is a subset of the Cartesian product 
A, XA, X: XAÁn 


As an example, the following three-dimensional unit sphere is a ternary relation on 
R7=RxRxXR. 


3 
S = To ER: $e < i} 


i=1 


2.3.1 Equivalence Relations 


Relations that exhibit and adhere to the three specific properties outlined in the forthcoming 
definition are of particular importance and offer substantial value within the context or 
domain in which they are applied or considered. 


Definition 2.6 Let A be a set and & bea relation on A. Then 


(i) & is said to be reflexive if for all x € A,x A x, and irreflexive otherwise. 
(ii) Z is said to be symmetric if for all x,y E€ A if xy, then y R x, and asymmetric 
otherwise. 
(iii) & is said to be transitive if for all x,y,z € A if x R y and y R z, then x Kz. 


2.3 Relations 


In light of Definition 2.6 and using (1.7), if & is a relation on a set A, then 


e & is not reflexive if there is some x € A such that x F x. 
e & is not symmetric if there are some x,y € A such that x Z y, andy F x. 
e A isnot transitive such that there are some x,y,z E A such thatx Z y and y A z butx F z. 


Example 2.8 Let A = {1, 2,3}, and define the following relations on A: 


= {(1, 1), (@, 2), (1, 2), (2, D}, 

= {0 1), @, 2), (3, 3), (L, 2), (2, 1), G, 3)}, 

{(1, 1), (2, 2), (3, 3), (L, 2), (2, 1), (1, 3), (3, D}, 

{(1, 1), (2, 2), (3, 3), A, 2), (2, 1)}, 

= {0 1), @, 2), (3, 3), (1, 2), (2, 1), (1, 3), (3, 1), 2, 3), (3, 2)}. 


Based on Definition 2.6, it can be seen that 


YVLQVa® 
I 


& is not reflexive because 3 & 3. 

S is reflexive, but not symmetric because 1 $ 3 and 3 $1. 

ZF is reflexive and symmetric, but not transitive because2 7 land1 JZ 3 but2 J3. 

U and Y are reflexive, symmetric, and transitive on A. The next definition calls % and 
Y equivalence relations. z 


Definition 2.7 A relation on a set A is called an equivalence relation on A if it is reflexive 
on A, symmetric, and transitive. 


Example 2.9 Among the three relations “=,” “<,” and “<” on N, “=” is the only equiva- 
lence relation on N. See Table 2.2. E 


Table 2.2 Some relations on the natural numbers. 


Relations on N: “=” “<” “<” 
Is the relation reflexive? Yes Yes No 
Is the relation symmetric? Yes No No 
Is the relation transitive? Yes Yes Yes 
Is it an equivalence relation? Yes No No 


An equivalence class is a set of elements that are considered to be similar or equivalent. 
We have the following definition. 


Definition 2.8 Let & be an equivalence relation on a set A. For x € A, the equivalence 
class of x determined by & is the set 
xX/R={yEA:xBy}. 


This is read “the class of x modulo &” or “x mod &.” When & is known from the context, 
equivalence class x/& is also denoted by [x]. 
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The set of all equivalence classes is called A module & and is denoted 


A/R = {x/R = [x] : x E€ A}. 


The following are examples of equivalence classes. 


Example 2.10 In Example 2.8, we have seen that @ = {(1, 1), (2, 2), (1, 2), (2, 1), (3, 3)} is 
an equivalence relation on A = {1, 2,3}. Itis also seen that 


1/@ =2/U = {1,2} and 3/% = {3}. 


Thus A/@ = {{1,2}, {3}}. 

In Example 2.8, we have also seen that Y is an equivalence relation on A. Finding 
1/V ,2/V ,3/V and A/Y is left as an exercise for the reader. a 
Example 2.11 The relation “o” on the set of all integers, Z, defined as x o y if and only 
if x? = y’, is an equivalence relation on Z. In this example, we have [1] = {1,-1}, [-2] = 
{—2,2}, and [0] = {0}. In fact, for any z € Z, we have [z] = {z, —z}. Thus 


Z/o = {0, {+1}, {+2}, {+3}, ...}. 


Example 2.12 (Congruence classes) 
Given an integer n > 1, called a modulus, two integers x and y are said to be congruent 
modulo n, denoted by x = y (mod n)orsimplyx =, y, ifnisa divisor of their difference (i.e., 
if there is an integer k such that a — b = kn). For example, 76 =,, 52 because 76 — 52 = 24 
which is 2 x 12. 

Congruence modulo n isa relation called a congruence relation. This congruence relation 
is an equivalence relation. This can be seen as follows. 


e Reflexivity: x = x (mod n). 
e Symmetry: x = y (mod n) ify =x (mod n) for all x,y, and n. 
e Transitivity: Ifx = y (mod n) andy =z(mod n), thenx = z(mod n). 


So, congruence modulo n is an equivalence relation, and the equivalence class of the 
integer a, denoted by [a],,, is the set 


[a], = {...,a — 2n,a — n,a,a + n,a + 2n, ...}. 


When the modulus n is known from the context that the congruence class is also denoted as 
[a]. This set, consisting of all the integers congruent to a modulo n, is called the congruence 
class, residue class, or simply residue of the integer a modulo n. E 


2.3.2 Ordering Relations 


In Table 2.2, we found that the relation “<” on N (or on any of the number systems Z or 
R) is not an equivalence relation because it is not symmetric. However, the reflexively and 
transitivity properties of “<” relation can be used in some applications such as the Big- 
Oh notation of algorithms which will be studied in Chapter 7. The following definition 


2.3 Relations 


introduces a property of relations that bridges the gap between symmetric and asymmetric 
relations. 


Definition 2.9 A relation & ona set A is called antisymmetric if, for all x,y E€ A, if x R y 
andy Z x, then x = y. 


For example, the relation “<” on R has the property that if x < y and y < x, then x = y. 
Therefore, this relation is antisymmetric. Other examples of antisymmetric relations will 
be presented after the next definition. 

Note that if a relation & is antisymmetric, then x # y and x # y implies that y F x. Note 
also that a relation may be antisymmetric and not symmetric, symmetric and not antisym- 
metric, both, or neither. Antisymmetry is one of the prerequisites for a partial ordering on 
a set A. We have the following definition. 


Definition 2.10 A relation & on a set A is called a partial order (or partial ordering) for 
A if Z is reflexive on A, antisymmetric and transitive. A set A with partial order & is called 
a partially ordered set, or poset. 


Besides the above example on the relation “<” which is a partial order for R, below are 
two other examples of partial orders taken from Rosen (2002). The first example shows that 
the relation “divides” on the set N of natural numbers is a partial order. Let D be the relation 
“divides” on N, then 1 7, 7H 7, 7 D 35, and 13 78. 


Example 2.13 The relation “divides” on N is reflexive on N because every natural number 
divides itself. Note that if a divides b and b divides c, then a divides c. This implies that the 
relation “divides” is transitive. Note also that if a divides b and b divides a, then a = b, 
which implies that this relation is antisymmetric. Thus, the relation “divides” is a partial 
order for N. E 


Example 2.14 LetA bea set. The set inclusion relation “C” on the powerset of A is reflex- 
ive on P(A) and transitive. Note that if A,B € P(A)and A C Band B C A, then A = B, which 
means that this relation is antisymmetric. Thus, the relation “C” is a partial order on the 
powerset P(A) for any set A. | 


Let & be arelation on a set A. Two elements of A are called comparable if they are related 
by &. For example, in the relation “divides” on N, 7 and 35 are comparable but 10 and 31 
are not comparable. Comparability is one of the prerequisites for a total ordering on a set 
A. We have the following definition. 


Definition 2.11 A partial ordering & on a set A is called a linear order (or total order) for 
A if for any two elements x ad y of A, either x A y or y &x, that is, x and y are comparable 
in &. Aset A with total order & is called a totally ordered set, linearly ordered set, or loset. 


For example, the relation “<” is a total order on the set N of natural numbers, but the 
“divides” relation is not a total order on N. The total ordering will be used in Chapter 9 to 
define the topological ordering of graphs. 
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2.4 Partitions 


Partitioning can be used to organize many things around us. The months of a year, for 
example, are partitioned in several ways: By the four seasons, by the month-lengths, by 
social/cultural events, etc.; see Figure 2.3. A partition of a set is a grouping of its elements 
into nonempty subsets, in such a way that every element is included in exactly one subset. 
Formally, we have the following definition followed by some examples of partitions. 


Definition 2.12 Let A be a nonempty set and A be a set of subsets of A. Then A is called 
a partition of A if the following conditions hold: 


(i) IfX € A, then X # Ø. 


(ii) UyeyX = A. 


(iii) IfX € AandY € A,thenX = Yorxn 


Y=@. 


Example 2.15 Here are four different partitions of N: 


{{1}, {2}, {3}, ...}. 
{Ay,A,,A,,A;}, where A; = {4k +i: k € N} fori = 0,1, 2,3. 


{E, O}, where E (respectively, O) is the set of even (respectively, odd) numbers. 


{N}. 


(a) 


Winter 


Fall 


Summer 


Spring 


© © 


(b) 


31 days 


28/29 days 


30 days 


0000 


O® ® © © © 


Figure 2.3 The year’s months (a), and their partitions by seasons (b) and by month-Lengths (c). 


2.5 Functions 


Note that the first and fourth partitions in the above example are the extremes in terms of 
the number of elements. In fact, for any nonempty set A, {{x} : x E€ A} and {A} are always 
partitions of A. 


Example 2.16 Here are three different partitions of the set R of real numbers: 


e {[nm,n+1): nE Z}. 
{ a Hee. 
e {Q,Q’}, where Q (resp., Q’) is the set of rational (resp., irrational) numbers. | 


A basic theorem of equivalence classes is the following. 


Theorem 2.1 The equivalence classes of any equivalence relation & in a set A from a 
partition of A and any partition of A determines an equivalence relation on A for which the 
sets in the partition are the equivalence classes. 


Proof: For the first part of the proof, we must show that the equivalence classes of & are 
nonempty, pairwise-disjoint sets whose union is A. Because & is reflexive, x € [x], and so 
the equivalence classes are nonempty; moreover, since every element x € A belongs to the 
equivalence class [x], the union of the equivalence classes is A. It remains to show that the 
equivalence classes are pairwise disjoint, that is, if two equivalence classes [x] and [y] have 
an element ain common, then they are in fact the same set. Suppose that x Za andy A a. By 
symmetry, a Z y, and by transitivity, x A y. Thus, for any arbitrary element b € [x], we have 
b Z x and, by transitivity, b Z y, and thus [x] € [y]. Similarly, [y] € [x], and thus [x] = [y]. 
For the second part of the proof, let % = {A;} bea partition of A, and define & = {(x,y) : 
there exists i such that x € A; and y € A;}. We claim that & is an equivalence relation on 
A. Reflexivity holds, since x € A; implies x 2 y. Symmetry holds, because if x Z y, then x 
and y are in the same set A,, and hence y A x. If x Ry andy & z, then all three elements are 
in the same set A,, and thus x & z and transitivity holds. To see that the sets in the partition 
are the equivalence classes of &, observe that if x € A;, then a € [x] implies a € A,, and 
a € A; implies a € [x]. The proof is complete. a 


Example 2.17 The set & = {B,, Bı, B,} is a partition of Z, where 


B = {3k : k€ Z}, B,={3k+1:keEZ}, andB,={3k+2:keZ}. 


The integers x and y are in the same set B; iff x = 3n + i and y = 3m + i for some integers n 
and m or, in other words, iff x — y is a multiple of 3. Thus, the equivalence relation associated 
with the partition & is the relation of congruence modulo 3 and each B; is the residue class 
of i modulo 3, for i = 0,1, 2. E 


2.5 Functions 


Functions are extremely important in mathematics and computer science. For example, but 
not limited to: In Section 3.1, sequences will be defined using functions. In Sections 5.1-5.4, 
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solutions of recurrences will be expressed as functions. In Chapter 7, functions are used to 
represent how long it takes a computer program to solve problems of a given size. 

The concept of a function is very old, but it is relatively recently that it has become 
standard to define a function as a relation with special properties. We have the following 
definition. 


Definition 2.13 A function (or mapping) from a set A to a set B is a relation f from A to 
B such that if (x,y) € f and (x,z) € f, then y = z. We write f : A > Band this is read “f is 
a function from A to B,” or “f maps A to B.” The set A is called the domain of f, and B is 
called the codomain of f. In the case where A = B, we say that f is a function on A. 


Note that, in Definition 2.13, no restriction is placed on the sets A and B. They may be 
sets of numbers, ordered pairs, functions, or even sets of sets of functions. 


Example 2.18 The following sets are relations from the set A = {1,2,3} to the set 
B= {a,b,c}. 


Z = {(1, a), (2, a), (2, b),(3,0)}, F = {(1, c), (2, ©), (3, a)}, 
$ = {(1, b), (2, a), (3, c)}, «U = {(1,a), (3, c)}. 
The relation & is not a function from A to B because (2, a) and (2, b) are distinct ordered 
pairs with the same first coordinates. See Figure 2.4. 
The relations $ and F are functions from A to B. The relation X is a function from {1, 3} 
to B because the domain of Y is not A, but the set {1,3}. E 


The vertical line test is a graphical method for testing whether the graph of a relation 
represents a graph of a function. The test states that if every vertical line intersects the graph 
of a relation at most once, then the relation is a function. 


Example 2.19 In Figure 2.5, we show two graphs. The graph shown on the left-hand side 
of Figure 2.5 is for the relation 2, = {(x,y) ER XR : xX? +y? = 1} with domain [-1, 1]. 
Since vertical segments intersect the graph of &, in two different points, Z, isnot a function 
from [—1, 1] to R. The graph shown on the right-hand side of Figure 2.5 is for the relation 
BR, = {(x, y) E [-az, 2] XR : y = sinx} with domain [—z, z]. Since every vertical segment 
intersects the graph of &, in at most one point, Æ, is a function from [—z, z] to R. a 


The function y = sinx whose graph is shown on the right-hand side of Figure 2.5 is the 
standard trigonometric sine function. There are different kinds of common functions, such 


A B A B A B A B 
if 4 
xX 
| \ AK 
R S A uU 


Figure 2.4 Different relations from A to B, some of them are functions from A to B. 


2.5 Functions 


y 
= 


Figure 2.5 The graphs of the circle x? + y? = 1 and the function y = sin x. 


as polynomials, trigonometric, exponential, logarithmic, etc. This can be found in any cal- 
culus textbook. In particular, a polynomial of one variable is a function that has the form 


P(x) = a,x" + Ay_yx" | +- + a,x? + ax + ao, 


where n is a nonnegative integer and the constant numbers ag, 4, Qz, ...,Q„ are the 
coefficients. 


Definition 2.14 Let a and b be elements of sets A and B, respectively. If f is a function 
from A to B and f(a) = b, we say that b is the image of a under f and a is a preimage of b 
under f. The range, or image, of f is the set of all elements of A. 


Figure 2.6 visually illustrates the difference between the terms range and codomain. 
When we define a function we specify its domain, its codomain, and the mapping of 
elements of the domain to elements in the codomain. 


Define the function f : R > R by f(x) = 1 + sinx. It y 
can be seen easily by looking at the graph of f(x) which 4 
is shown to the right that: i l+ sin x 


1+sina 


- The domain off is R. 
- The range of f is [0, 2]. 
- The codomain off is R. i 


> xX 
Since f(a) = 1 + sina, the value 1 + sin a is the image = d 
of a under f, and the value a is a preimage of 1 + sina 
under f. 
Figure 2.6 Range versus 
codomain. Codomain 


Domain Range 
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TTT oT ooo?“ Figure 2.7 The graph of the 
400 L integer-valued cubic function 
y =x? with domain [—8, 8] n Z. 


Two functions are equal when they have the same domain, and map each element of 
their common domain to the same image. A function is called real-valued if its codomain 
is R, and it is called integer-valued if its codomain is Z. Figure 2.7 shows an example of an 
integer-valued function. 

Another common example of integer-valued functions is the floor function. This func- 
tion, also known the greatest integer function, has domain R and codomain Z. It maps 
each real number x to the greatest integer n such that n < x. We use the notation |x] for 
this function. Specifically, |z] = 3 and |-z] = —4. The ceiling function is also an example 
of integer-valued functions. This function, also known the smallest integer function, has 
domain R and codomain Z. It maps each real number x to the smallest integer n such that 
n > x. We use the notation [x] for this function. Specifically, [z] = 4 and [~z] = —3. 


2.5.1 Surjections 


A function is said to be a surjection or an onto function if its range and codomain are the 
same. For example, the two functions 


f:R—-R, where f(x) = |x], 
g: R> R,, where g(x) = |x], 


are equal. Here R, denotes the set of nonnegative real numbers. The functions f and g have 
the same range, which is R,, but f maps to R while g maps to R,. Therefore, f is not onto 
while g is onto. We formally have the following definition. 


Definition 2.15 A function f : A > B is said to be onto, or a surjection, if every element 
in B has a preimage, that is, for every element b € B there is an element a € A with f(a) = b. 


Example 2.20 To show that the function f : R > R, where f(x) = 2x +1, is onto, we 
must show that for every t € R, there exists s € R such that f(s) = t. Let t € R, and choose 
s = (t — 1)/2. Then f(s) = 2s + 1 = 2((t — 1)/2) + 1 = t. Thus, f is onto. E 


2.5 Functions 


In light of Definition 2.15, to show that a function f : A —> Bis not a surjection, we need 
to find a particular b € B such that f(a) + b for all a € A. We have the following example. 


Example 2.21 Toshow that the functiong : R > R, where g(x) = x”, is not onto, we must 
find a value t € R that has no preimage in R. Let t = —1. Since x? > 0 for every x € R, there 
is nox E R such that g(x) = —1. Thus g is not onto. a 


2.5.2 Injections 


A function f is said to be an injection or a one-to-one function if no two elements in the 
domain of f with equal images. For example, the two functions 

f:R—-R, where f(x) = |x], 

g: R} >R, where g(x) = |x], 


are different. The functions f and g have the same range, which is R,, but the domain of 
f is R while the domain of g is R,. Note that f(1) = f(—1) = 1, which means that there are 
two different numbers —1,1 € R with equal images under f. In contrast, no two different 
numbers in R, with equal images under g. Therefore, f is not one-to-one while g is one-to- 
one. We formally have the following definition. 


Definition 2.16 A function f : A > B is said to be one-to-one, or an injection, if f(a) = 
f(b) implies that a = b for all a and b in the domain of f. 


Example 2.22 Determine whether each of the following functions is one-to-one on its 
domain. Justify your answer. 


(i) fœ) =x- 8. (ii) g(x) = x x Gii) h(x) =x +3. 
x — 
Solution 


(i) Clearly, the domain off is R. Let a,b € R, then 


f@=fb) => @&-8=b?-8 = =b. 


It follows from this that a = b. Thus, f is one-to-one. 
(ii) The domain of g is R — {8}. Let a,b € R — {8}, then 
a b 
g(a) = g(b) = “= 
It follows from this that a = b. Thus, g is one-to-one. 
(iii) Clearly, the domain of h is R. Let a,b € R, then 


=> ab- ŝa = ab — 8b = 8a= 8b. 


h(a) =h(b) = at +3 =b +3 = at =b. 


It does not follow from this that a = b. In fact, this failed “proof” suggests a way to find 
real numbers with equal images. Indeed, h(1) = 4 = h(—1) while 1 4 —1. This shows 
that h is not one-to-one. E 
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In light of Definition 2.16, to show that a function f : A > Bis not an injection, we need 
to find particular elements a, b € A such that a + b and f(a) = f(b). We have the following 
example. 


Example 2.23 Which one of the following functions is not an injection? 


(i) Aæ) = sinx for x € [0, z]. (iii) hœ) =x? + 1 for x > 0. 

Gi) hœ = |x + 1| for x > 0. (iv) fŒ) = cosx for x € [—z, 0]. 

Solution 

The correct answer is (i). In fact, the function f(x) = sinx, x € [0, z], is not an injection 
because f,(2/3) = f,(2"/3) = 3/2 while 1/3 £ 27/3. a 


The horizontal line test is a graphical method for testing whether the graph of a function 
represents a graph of an injection. The test states that if every horizontal line intersects the 
graph of a function at most once, then the function is an injection. 


Example 2.24 In Figure 2.8, we show two graphs. The graph shown on the left-hand side 
of Figure 2.8 is for the trigonometric function f(x) = cosx with domain [—z, z]. Since hori- 
zontal segments intersect this graph in two different point, f(x) with domain [—z, z] is not 
an injection. The graph shown on the right-hand side of Figure 2.8 is for the same function, 
f(x) = cosx, but with domain [0, z]. Since every horizontal segment intersects this graph in 
at most one point, f(x) with domain [0, z] is an injection. a 


2.5.3 Bijections 


Bijective functions are essential to many areas of mathematics. For example, in Section 4.2, 
we will see that bijections arise in the definition of graph isomorphism. 


Definition 2.17 A function is said to be a one-to-one correspondence, or a bijection, if it 
is both injection and surjection. 


The function f : R} > R, defined by g(x) = |x| is an example of a bijection. See also 
Figure 2.9 which shows examples of different types of functions. Only the function shown 
on the right-hand side of Figure 2.9 is bijection. 


y y Figure 2.8 The graphs of the function y = cos x 
with two different domains. 


y=cosx 


y=cosx 
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Surjection Injection Neither Bijection 


Figure 2.9 Examples of different types of functions. 


Exercises 


2.1 Choose the correct answer for each of the following multiple-choice question- 
s/items. 
(a) One of the following statements is true about mathematical induction proofs, 
which is 
(i) Every mathematical induction proof is a two-part proof, and only the first 
part is necessary. 
(ii) Every mathematical induction proof is a two-part proof, and only the sec- 
ond part is necessary. 
(iii) Every mathematical induction proof is a two-part proof, and both parts are 
absolutely necessary. 
(iv) None of the above is true. 
(b) Ifinduction on n is used to prove that 


n 
py = Zna +1)(2n +1)(3n?+3n-1), forn EN, 
i=1 


then the base case (basis step) is 
© Yin l* = 510 +D2+DG+3-D, nen. 
m n a1 _ 
Gi). >= 441d +D@2+DG+3-1), nEN. 
(ii) Dia = 510 + D2+13+3-D). 
(iv) none of the above. 


(c) Ifinduction on n is used to prove that 


n 
py = a(n +1)’, forneN, 
Ei 


then the inductive step shows that 
O E P = ik +1? + 2). 
G) Dare = ik + 12k + 2} — Zk b= 1h (k + 1). 
Gi) YB = ikk +1)? —> Yi B= ik?(k + 1). 
(iv) DB = ik?(k +1} —> re B= i(k + 1)?(k + 2). 


72 


2 Set-Theoretic Structures 


2.2 


2.3 


2.4 


(d) If we list all of the proper subsets of the set {{@}}, we get 
(i) Ø. (iii) {Ø}. 
(ii) Ø, {Ø}. (iv) no proper subsets. 


(e) Let D be a domain and X = {x € D : P(x)}. Only one of the following statements 
is false, which is 


(i) ae X — Pa). (iii) -P(a) — a ¢ X. 
(ii) Pa) — a EX. (iv) Pla) —> a €D. 
(£) The binary relation {(a,a), (b, a), (b,b), (b,c), (b, d), (c,a),(c,b)} on the set 
{a,b,c} is 


(i) reflective, symmetric, and transitive. 
(ii) irreflexive, symmetric, and transitive. 
(iii) irreflexive and antisymmetric. 
(iv) neither reflective, nor irreflexive but transitive. 
(g) The relation x & y if |x| = |y| is 


(i) transitive and symmetric. (iii) reflexive and asymmetric. 
(ii) reflexive, symmetric, and (iv) irreflexive, antisymmetric, and 
transitive. transitive. 


(h) Let A = {1,2,3,4,5, 6,7}. Which one of the following is not a partition of A? 


(i) {{1, 2, 5}, {3,6}, {4,7}}. (iii) {{1, 5, 7}, {3,4}, {2,5,6}}. 
(ii) {{1, 2, 5,7}, {3}, {4, 6}}. (iv) {{1,2,3,4,5,6,7}}. 
(i) Which one of the following relations is not a function? 
G) {@y)E RXR: y= |x}}. (iii) {x,y ERxXR: y=x}. 
(ii) {%y) E RXR: x= |yl}. (iv) {@%y) E RXR : x? =y}. 
(j) Which one of the following functions is not a bijection? 
G) {~%y)ERXR: y=x}. Gii) {@,y) E R} xR: y= |x|}. 
Gi) {%&y) ERXR: y=x3}. (iv) {~%y) ERXR: y=x?}. 


Archimedean principle states that for all natural numbers n and m, there exists a 
natural number s such that m < sn. This principle can be proven by induction on 
n. To prove this for n = 1, choose s to be m+ 1, then m < m + 1 = sn. So the base 
step is true. Assuming the principle is true when n = k for some k € N, complete 
the induction proof by establishing the inductive step. 


Use mathematical induction to prove that, for all n € N, we have 


5e _ nnt+1j2n+1) 
i=1 E 6 l 


Use mathematical induction to prove that the cardinality of the powerset of a 
finite set A is equal to 2” if the cardinality of A is n. Show all the steps in the 
proof. 
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2.5 Use the induction method to show that the recurrence 


Tn) = Í, itas; 
3T(n—1)+4, ifn = 2,3,4,..., 


has the solution given in (2.6). 


2.6 LetA, B, and C be any three sets. Decide whether each of the following implications 
true or false? And if it is false, give an example that shows that. 


(a) 4 EBABEO—(4E0. MAUCCBUC —— ACB. 


2.7 Find the number of subsets of the given set. Justify your answer. 
(a) S = {fundamentals, discrete, structures, combinatorics, optimization }. 
(b) T = {n EN: nisan even number between 21 and 41}. 


2.8 LetS = {q, T, s, t, u, V, W, X, y, zZ} be a space or universe of three sets A, B and C, where 
A= {q, S, u, w, y}, B = {q,s, y, Z}, and C = {v, w, x, y, z}. Identify the set (A U C’) N 
P' by listing its members in set braces. 


2.9 Let & be the relation on a set of all people defined as x & y if and only if x and y have 
the same birthday (out of 365 possible birthdays). Show that & is an equivalence 
relation. How many equivalence classes does & have? 


2.10 Show that the exponential functionf : R — [0, co) defined by f(x) = e” isa bijection. 
Is the exponential function g : R > R defined by g(x) = e a bijection? Justify your 
answer. 


Notes and Sources 


The history of set theory can be traced to the late 19th century, with significant contributions 
from mathematicians like Georg Cantor, who is often regarded as the founder of modern set 
theory. In the 1870s, he introduced the concept ofa set as a collection of distinct objects and 
developed a formal theory of infinite sets. His work on the theory of transfinite numbers 
and the notion of different “infinities” revolutionized the field of mathematics. Cantor’s 
book Cantor (1883) formed the basis for the growth and formalization of set theory in the 
20th century. 

This chapter introduced several set-theoretic concepts that serve as the foundation for 
various mathematical structures and algorithms in computer science. To be more precise, 
the chapter covered mathematical induction, sets along with their properties, relations, 
equivalence relations, ordering relations, partition, and functions along with their char- 
acteristics. The content included in this chapter not only provided readers with essential 
mathematical tools, but also prepared them for more advanced studies in mathematics and 
computer science. 
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As we conclude this chapter, it is worth noting that the cited references and others, such as 
Cusack and Santos (2021); Pinter (2014); Mott et al. (1986); Joshi (1989); Lipschutz (c1981); 
Lin and Lin (1985); Kunen (1983); Fraenkel et al. (1973); Jech (1997); Schwartz et al. (2011); 
Scheurer (1994); Moschovakis (2006), and Knuth (1997), also serve as valuable sources of 
information pertaining to the subject matter covered in this chapter. The code that created 
Figure 2.1 is due to StackExchange (2013). We used and modified a code due to StackEx- 
change (2020) to draw Venn diagrams (Figure 2.2). Exercises 2.2 and 2.6 are due to Smith 
et al. (2014). 
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The first two sections of this chapter study analytic concepts, such as sequences, sum- 
mations, and series, which are used extensively in solving recurrences in Chapter 5. The 
last three sections of this chapter study algebraic and geometric structures, such as sub- 
spaces, bases, convex sets, convex cones, convex hulls, and polyhedra, which are, together 
with Farkas’ lemma of Section 3.5, used extensively in solving linear programming in 
Chapter 10. 


3.1 Sequences 


A sequence of real numbers is a function a : N > R. The values a(1), a(2), ..., which for 
simplicity are usually written as a,,a,,..., are called the terms of the sequence. In particu- 
lar, a,, is called the nth term of a. The sequence is often written as {a}, a3, a3, -.. }, {a,}7,, 
or even more simply {a, }. 


Example 3.1 The first five terms of the sequence {a,,} = { aa) are 


a, = 1/3,a, = 2/4,a, = 3/5,a, = 4/6, and a, = 5/7. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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Example 3.2 Itis not hard to see that the nth term of the sequence that has the five given 
terms b, = —1/4, b, = 2/9, b, = —3/16, b, = 4/25 and b; = —5/36 is 
n 


WAD aie 1 


A sequence can be infinite, as in Examples 3.1 and 3.2, or finite if it has a limited number 
of terms. 

A sequence can also defined by a recurrence relation, which expresses each term as a 
combination of the previous terms. For example, the Fibonacci Sequence,! So-fi-fa: ++» is 
defined by the recurrence relation 


Sn = fa +fa-2; for n = 2,3,4,..., where fp =0andf, =1. 
Using the recurrence formula, we have 


h=afth=ait+0=1, fp=f,+f=3+2=5, 
Íz htf Lad 2, Jo fs+tfa 5+3 8, 
fa h +h 2+1 3, Ja Je +f 8+5 13. 


Solving recurrences is an important subject that will be studied exclusively in Chapter 5. 

A sequence {a,,} is called bounded above if there is a number M such that a„ < M for all 
n. Here, M is called an upper bound for {a,,}. A sequence {a,,} is called bounded below if 
there isa number m such that a„ > m for all n. Here, m is called a lower bound for {a,,}. The 
sequence is called bounded if it is both bounded above and bounded below. For example, 
since 0 < 2 < 1 for every n, the sequence { 2 } is bounded below by 0 and is bounded above 
by 1. 

Now, we study the limits of sequences. Because we are only concerned with limits 
involving infinity, we introduce the following definition. 


Definition 3.1 (Limit) Leta, : N > RbeasequenceandL € R. Itis said that the limit 
of a,,, as n approaches infinity, is L, and written lim, _,,,a,, = L, if for every real number 
e > 0, there exists ng E€ N such that |a, — L| < e whenever n > no. 


The following example illustrates how Definition 3.1 is applied. 


Example 3.3 Prove that lim,,_,,,1/n = 0. 
Solution 
Let e > 0 and choose an integer nọ > 1/e. Then 1/ny < e. Now, for any n > ng, we have 
paten 
n w 

and so 

1-o|= tol ad eh 

n n| n` No 
By Definition 3.1, we conclude that lim, _,,,1/n = 0. E 


1 The Fibonacci sequence is named after the Italian mathematician, Leonardo Fibonacci, who was born in 
1170. 


3.1 Sequences 


Example 3.4 400 
As n grows larger and larger 
without limit, the output of the 
mathematical function expressed 300 + = 
as n? also increases without 
bounds. In simpler terms, the 
values of the quadratic function ~“ 200 
n? become progressively larger, 
without any upper bound or limit, 
as n > oo. See the discrete graph 100 F -J 
shown to the right. Therefore, the 


ii Laettt ll 
5 


lim n? 0 


no 


10 15 20 


does not exist. ü 


A sequence that has a limit is said to be convergent. A sequence that has no limit is said 
to be divergent. For instance, the sequence {1/n} in Example 3.3 is convergent to 0 and the 
sequence {n? + 4} in Example 3.4 is divergent. 


Theorem 3.1 Every convergent sequence is bounded. 
Proof: Assume that a,, > L. Choose any positive number, say 1, and use it as e. Then we 
can see that there exists a positive integer ny) such that 
ja, -L| <1 forall n > nọ. 
Note that 
la,| = ILI < llan] — ILI] < la, = LI. 
It follows that 
la,| < |IL| +1 forall n > ng. 
Consequently, we have 
a, < max {]a;l, lasl, -> lan -1l E +1} forall n. 
This proves that {a,,} is bounded and hence proves the theorem. a 


The truth of the following corollary follows from the contrapositive law (see Table 1.8) 
and Theorem 3.1. 


Corollary 3.1 Every unbounded sequence is divergent. 


For example, the sequences 
{a,} =n? and {b,} =ne" 


are unbounded. Therefore, each of these sequences is divergent. 
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Using Definition 3.1, we can prove the following theorems. 


Theorem 3.2 Let lim a, =Landlim b, = M, where L and M are finite real num- 


bers. Then i a 

(i) lim, _,,.ka, = kL for any k E€ R. (iii) limps% (anbn) = LM. 

(ii) lim, ,..(@, + b,) = LM. (iv) IfM £0, impo z = x. 

Theorem 3.3 Let k be a constant. The following limits hold. 

Gi) lim, ,.,.k =k. (iii) Ifk > 0, lim,,_,,,log, n = co 
oo, if k > 0; oo, ifk>1; 


(ii) lim, ,.2* = { (iv) lim," = { 


0, ifk <0. 0, if0<k<l1. 


noon 


Theorem 3.4 Let {a,,} be a sequence. If lim, ,,,a,, = œ, then lim, oo = 0. 


A well-known and very useful theorem for computing limits is the following. 
Theorem 3.5 (L’Hospital’s rule at infinity) Let f(n) and g(n) be two real-valued dif- 
ferentiable functions on N. If 
limf(n) = limg(n) = 0 or limf(n) = limg(n) = 
then 


tim? = gi 
Sgm Moga) 


This version of L’Hospital’s rule is restricted to the limits where the variable n approaches 
infinity since these are only limits of interest in our context. 


Example 3.5 Test the following sequences for convergence. 


O {a,} = {=z}. (ii) {b,} = {2} Gii) {c,} = {=}. 


Solution 
(i) Both numerator and denominator tend to œ as n > oo. Using L’Hospital’s rule, we 
have 
1/n 
lima, = fing = = lim iL = lim =0. 
n>% n>% nN n>% 2n n>% 2n2 


Therefore, {a,,} converges to 0. 
(ii) Both numerator and denominator tend to œ as n > oo. Applying L’Hospital’s rule 
twice, we have 


2 
lima, = = lim” = ace = lim Ż =0. 


n>% n>0o0 e” n>o e” n>o et 


Therefore, {b,,} converges to 0. 
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(iii) Both the numerator and denominator tend to 0 as n > oo. Using L’Hospital’s rule, we 


have 
zjn erl” -r n2 
lima, = lim — = ii a E TE 
n> n>% 1/n n>% (-1/n?) n>% 
Therefore, {c,,} converges to z. a 


3.2 Summations and Series 


Summations and series are used to describe algebraic patterns. Series are of great interest 
because computers and smartphones use them internally to calculate many functions. 
If we add the terms of an infinite sequence {a,}/° ,, we get 


Ap +A, +a,+---+a,+---. (3.1) 
We write 

co 

Da (3.2) 

k=0 


(read “the sum of a sub k from k equals 0 to k equals infinity”) to indicate the sum (3.1). 
The infinite sum (3.2) is called an infinite series. The corresponding sequence {s,,} defined 
by the finite sum 


n 
Sy = My +0, +0, +--+ +4, = Dia, 
k=0 


is called the sequence of partial sums of the series (3.2). 


Definition 3.2 Given the infinite series {a} Po: If the sequence of partial sums {s,,}, with 
Sn = Èro 4p converges to a finite limit L, then the series $ ¿o a; is said to converge to L, 
written as 


5G, =L. 
k=0 


The number L is called the sum of the series. If the sequence of partial sums diverges, then 
the series $ go a; diverges. 


Using Definition 3.2, we can prove the following theorem. 


Theorem 3.6 If ap and $ipo b, are convergent series, then their sum, difference, 
and scalar multiplication by any constant are convergent series. Moreover, 


(i) yn + vy = Sa + b). (ii) yaa, — aS a Va E R. 
k=0 k=0 k=0 k=0 


k=0 


It is important to point out that the convergence or divergence of an infinite series is not 
affected by where you start the summation because the limit of the sequence of partial sums 
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does not depend on where you begin the summation. Therefore, $o, converges if and 
only if $ gm ay converges, for any positive integer m. 


Example 3.6 Test the following series for convergence. If it converges, find its sum. 


Lae 

Solution 

Note that 
1 _(k+1I)-k 1 1 
k(k+1) k(k+1) k k+l 


It follows that the sequence of partial sums is 
A/A 1 
Sre = D~ za 
ee eee 
1 2 2 3 n-l n n n+l 


DM aod hea eel | ei 
2 2 3 n-1 n n n+l n+1 
Now, as n > œ, S, > 1. This means that the series converges to 1. Therefore 


ys 
Aik (Kk +1) 


This answers the question raised in the example. a 


Infinite series with the property that its terms can be arranged in pairs with opposite signs, 
except for the first and last terms, are called telescoping series. For instance, the infinite 
series given in Example 3.6 is a telescoping series. 

The following theorem states that the kth term of a convergent series tends to 0. 


Theorem 3.7 


foe} 


If Xa, converges, then a, > 0 as k > œ. 
k=m 


Proof: Let 


n 
py 
k=m 


Since the series $, a, converges, its sequence of partial sums, {s,,}, converges to some 
number L. That is, s, > L. Hence, s,_, > L as well. Since a, =s, — 5, _,, we have a, > 
L -L = 0. A change in notation gives a, — 0. The proof is complete. E 


3.2 Summations and Series 


The truth of the following corollary follows from the contrapositive law (see Table 1.8) 
and Theorem 3.7. 


Corollary 3.2 (The divergence test) 
Ifa, » 0 as k > œ, then Za, diverges. 
k=m 


For example, since lim,_,,,(k/(k + 1)) = 1 # 0, using the divergence test, we conclude that 
the series X go 4 diverges. 

We now introduce the geometric series, which is the most famous convergent series. The 
series is first presented in its finite version. 


Theorem 3.8 (The finite geometric series) Letx € R — {0}, and n and m be integers 
such that 0 < m < n. Then 
x” — x”+! 


Yate { Tor eee (3.3) 
k=m 


n—-m+1 ifx=1. 


Proof: Let 


n 

_ k 

s= Ye. 
k=m 


To compute s,,, first multiply both sides of the equality by r and then manipulate the result- 
ing sum as follows. 


n n n+1 n 
XS, = ae = ae = > x = x +O =y"), 
k=m k=m j 


It follows that 
XS_ = Sp + OH — x”), 


Solving for s,, shows that if x + 1, then 


xm — xt 
8, = ——— 
i 1-x 
Ifx = 1, then s„ = )y_,, 1 =n—m-+ 1as desired. This proves the theorem. a 


Corollary 3.3 Letx € R, and n and m be integers such that 0 < m < n. Then 


Ès =n-m+ Df, (3.4) 


k=m 


where f(-) is a real-valued function that does not depend on k. 


The following example is a direct application of (3.3) and (3.4). 
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Example 3.7 Find 

TENG 

© z). (ii) 
AG) i AAO 


Solution 
(i) Using (3.3), we have 


( 3 j” 7 
? 3k 7 ee 9 
3 4 4 3 
2 —— =3(1-(2) ) x~ 2.775. 
k=l 371 
(ii) This is an example of a double summation. To evaluate this, first compute the inner 
summation and then continue by computing the outer summation as follows. 


AE j -X (0- 0+ 0(5 J) = 1091 (3)' = covers) = 202, 


where we used (3.4) to obtain the first equality and used item (i) to obtain the approxi- 
mate equality. a 


Now, we state and prove the infinite version of the geometric series. 
Theorem 3.9 ae infinite geometric series) Ifx € R, then 


if |x| <1, 


Èr- =J1-x (3.5) 


diverges if |x| > 1. 


Proof: Let 


n 

a2 k 

5, = Ye. 
k=m 


We break the proof into two cases: 


Case 1. When |x| = 1. If x = 1, then s, =n—m-+1 and hence {s,,} diverges as n > oo. 
Therefore $, x" diverges. If x = —1, then x* = (-1)‘ + 0 as k > œ and hence by the 
divergence test, the series X2, x* diverges. 

Case 2. When |x| # 1, the same steps followed in the proof of Theorem 3.8 can be used to 
prove that 

x1 — xm 


i x-1 
If |x| > 1, then {x"*! } is unbounded and hence {s,,} diverges as n > oo. Therefore, X pom xk 
diverges. If |x| < 1, then {x"t'} 0 as n > oo, and hence 


xm 
S, > — 
" 1=x 
Therefore 
mo m m 
. . Xx xX 
X = lims, = lim —— = —. 
n>% n>%] -x 1-x 
k=m 


This proves the theorem. a 


3.2 Summations and Series 


The following example is a direct application of (3.5). 


Example 3.8 Test the series for convergence. If it converges, find its sum. 


a we 5\k a wa 5\3-* 
© XG): o 23) 


Solution 
(i) The series diverges as 5/4 > 1. 
(ii) We have 


ee 3-k See k 3 /5)° 
Di). ea) RG eae 


Thus the series converges to 5. a 


We can use the geometric series to derive an explicit formula, also called a closed form, 
for some series. 


Example 3.9 Use the geometric series to derive a closed form for the following series. 
foe} 
Len, |x| < 1. 
k=0 


Solution 
Since |- x| = |x| < 1, using (3.5), we have 


Saye = yk 1 _ 1 
2 1) E D = Io 


E 
The proof of the following theorem is left as an exercise for the reader. 
Theorem 3.10 Let f(x) = Ejo apt and g(x) = Xo bpd. Then 
m+n k 
fase =>) ( ar] ae (3.6) 
k=0 \r=0 


where we use the convention that a; = 0 for all integers i > m and b, = 0 for all integers 
j>n. 


The following is the infinite version of Theorem 3.10. 


Theorem 3.11 Let f(x) = Xp oax" and g(x) = Y2.9b,x*. Then 


00 k 
foga) = £ ( ab) x (3.7) 
r=0 


k=0 


The following example shows how Theorem 3.11 is applied. 


85 


86 


3 Analytic and Algebraic Structures 


Example 3.10 Find a series expansion for the expression 1/(1 — x)’, |x| < 1. 


Solution 
Since |x| < 1, we have 


n 
1 
Ese 


Using Theorem 3.11, we have 


co k œ 
l 
— = 1]x*= A + 1)x*, 
This answers the question raised in -a example. E 


Another way to approach the series expansion for the expressions, such as that in 
Example 3.10, is to use Theorem 3.12. The proof of the following theorem can be found in 
any standard calculus textbook, see for example (Salas et al., 2006, Theorem 12.9.1). 


Theorem 3.12 If a,x converges on (—c,c), then Dro Lax) also converges on 
(—c, c). Moreover, if 


fx) = Faxi forall x € (—c,c), 


k=0 
then f is differentiable on (—c, c) and 


Fos ye —(a,x*) for all x € (-c,c). 


aod 


The result in Example 3.10 can be derived from Theorem 3.12 by differentiation as 
follows. 


d alg : 
Tilil e) = Lion- Der -F+ 


It is important to point out that the above differentiation is valid provided that |x| < 1. 

Table 3.1 provides closed forms for commonly occurring summations. Some summa- 
tions given in the left-hand column of Table 3.1 were proven in Section 2.1 by induction 
(see Corollary 2.1 and Exercises 2.1 and 2.3). 


Table 3.1 Some useful summation formulas. 


Sum Closed form Sum Closed form 
n n = yntl 
>A n A x#1 tee 
k=1 k=0 1-x 
n 4 1 foe} 
Yk my) Yx, [xl <1 a 
k=1 2 k=0 1-x 
= +1)(2n+1 = 
ye eee) Fep, bl <1 ae 
6 = 1+x 
=0 
= n’(n+1)? 1 
E — k+1)x*, |x| <1 —— 
pa E Dy x“, Ixl TEE 


3.3 Matrices, Subspaces, and Bases 
3.3 Matrices, Subspaces, and Bases 


In this section, we review some notions and concepts from elementary linear algebra. 


3.3.1 Matrices 


Throughout this and other subsequent chapters, we will use vectors and matrices. A matrix 


of dimension n x mis an array of numbers a;;: 


Ay, Ayn = Am 
A= |02 a2 + Gam 
An U2 ETE Anm 


We assume that the entries of A are real. A row vector is a matrix with n = 1, and a column 
vector is a matrix with m = 1. The word vector will always mean column vector unless the 
contrary is explicitly stated. So, a vector of dimension n is an array of numbers x;: 


x} 


Scalars will be always denoted by lowercase characters, vectors will be always denoted 
by lowercase boldface characters, and matrices will be always denoted by uppercase 
characters. 

We use R” to denote the set of all nth-dimensional vectors. The standard inner product 
of R” is defined as 


n 
(x,y) Ê x'y = X xyi 
i=l 
forx,y € R”. Herex! denotes the transpose of the vector x. The Euclidean norm (also called 


the 2-norm) of x € R” is denoted as ||-|| and is defined to be the square root of the inner 
product of a vector with itself. That is 


ixl] = Vxtx = 


forx € R”. 
The transpose A! of an n x m matrix A is the m x n matrix 


G Ay. Any 
Al | Fae 12 ee m 
Aim Grn +++ nm 


A square matrix is a matrix with the same number of rows and columns. A square matrix 
A is called symmetric if it is equal to its own transpose matrix, that is, A = AT. We use I to 
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denote the identity matrix, which is a square matrix whose diagonal entries are ones and 
its off-diagonal entries are zeros. 

A square matrix is called a diagonal matrix if all of the entries off the main diagonal are 
zero. That is, a square matrix D is diagonal if d, = 0 whenever i + j. A square matrix is 
called upper triangular if all the entries above the main diagonal are zero. Therefore, an 
upper triangular matrix has the form: 


xxx xX 
|x xxx 
xxx 
0 lxx 
x 


Likewise, a square matrix is called lower triangular ifall the entries above the main diagonal 
are zero. 

Let A be a square matrix. If there exists a square matrix B of the same dimensions satis- 
fying AB = BA = I, we say that A is invertible or nonsingular. Such a matrix B, called the 
inverse of A, is unique and is denoted by A~!. The matrix A is called an orthogonal matrix 
if ATA = I. In other words, a square matrix A is orthogonal if its transpose is equal to its 
inverse matrix, that is, AT = A~!. 

The determinant is a function whose input is a square matrix A and whose output is a 
number. We use the notation det (A) to denote the determinant of a square matrix A. The 
reader can consult any linear algebra textbook, such as Anton and Rorres (2014), to see how 
to compute det (-). 

Ifa system oflinear equations Ax = b has at least one solution, it is said to be consistent. A 
consistent system has either exactly one solution or an infinite number of solutions. There- 
fore, for a system of linear equations, we have three possibilities: The system has a unique 
solution, it has infinitely many solutions, or it is inconsistent. We give, without proof, the 
following standard result in this context. For a proof, see for example (Anton and Rorres, 
2014, Theorem 4.8.8). 


Theorem 3.13 Let A be a square matrix. Then, the following statements are equivalent: 


(a) The matrix A is invertible. 

(b) The matrix A’ is invertible. 

(c) The determinant of A is nonzero. 

(d) The rows of A are linearly independent. 

(e) The columns of A are linearly independent. 

(£) For every vector b, the linear system Ax = b has a unique solution. 

(g) There exists some vector b such that the linear system Ax = b has a unique solution. 
(h) det(A) #0. 


In this section (and throughout the book), we use R™ to denote the set of all real matri- 
ces of dimension n x m. 


Definition 3.3 A matrix A € R”" is called positive semidefinite (respectively, positive 
definite) if it is symmetric and x'Ax > 0 for all x € R” (respectively, x'Ax > 0 for all 
x € R” — {0}). 


3.3 Matrices, Subspaces, and Bases 


The eigenvalues of a matrix are the roots of its characteristic polynomial (for more detail, 
consult any linear algebra text; for example, refer to Renardy (1996)). As an alternative to 
Definition 3.3, we say that a square matrix is positive semidefinite (respectively, positive 
definite) if it is symmetric and all its eigenvalues are nonnegative (respectively, positive). 
Note that every positive definite matrix is invertible. 


3.3.2 Subspaces and Bases 


Subspaces and bases are introduced in this part. We have the following definition. 


Definition 3.4 AsetS C R” isasubspace ifax + by € Sforeveryx,y € Sandeverya,b € 
R. If, in addition, S # R”, we say that S is a proper subspace. 


Note that the zero vector, 0, must belong to every subspace (take a= b = 0). As an 
example, every line passing the origin is a subspace of R?. As another example, every plane 
passing the origin is a subspace of R?. 


Definition 3.5 We say the vectors x, x®, ...,x°” © R” are linearly dependent if there 
exists a}, d2, ... , Am E R, not all of them zero, such that pn a,x = 0; otherwise, they are 
called linearly independent. 


For example, clearly the two vectors (1, 0)' and (0,1)' are linearly independent in R?. For 
another example, it is also clear that the three vectors (1, 0,0)", (0, 1,0)", and (0,0,1)' are 
linearly independent in R°. Note that the maximum number of linearly independent points 
in R” isn. 

As a basic fact in linear algebra, the maximum number of linearly independent rows of a 
matrix A is equal to the maximum number of linearly independent columns of A. 


Definition 3.6 The rank of a matrix A, denoted by rank(A), is the maximum number of 
linearly independent rows (columns) of A. 


Definition 3.7 The span of a finite number of vectors x,x®,...,x in R” is the 
subspace of R” defined as the set of all vectors y of the form y = Ð; a,x, where 
a1, Q2, -.. , Am E R. Any such vector y is called a linear combination of x® ,x®, ... ,x™. 


Definition 3.8 A basis of a nonzero subspace S C R” is a collection of vectors that are 
linearly independent and whose span is S. The dimension of a subspace is the number of 
vectors in any basis for the subspace.” 


For example, the rank of the 3 x 3 identity matrix is 3. The space R? has {(1,0,0)', 
(0, 1,0)", (0,0, 1)" } as a span. This spanning set is also a basis. Therefore, the dimension of 
R? is 3. In general, the dimension of R” is n. 


2 Every basis of a given subspace has the same number of vectors and this number is its dimension. 
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Definition 3.9 Let S be a subspace of R” and x, É S be a vector. The set Sọ =x) + S = 
{xo +x : x € S} is called an affine subspace parallel to S. 


It is not hard to see that the dimension of Sọ is equal to the dimension of the subspace S. If 
Sis an m-dimensional subspace of R” with m < n, there will be n — m linearly independent 
vectors orthogonal to S. The set of such orthogonal vectors is indeed a subspace of R”. 
Definition 3.10 If S C R” isa subspace, then the subspace 

S+ 2 {xeER": x'y=0 fory €S} 


is called the orthogonal subspace of S. 
We give, without proof, the following standard result in this context. 


Proposition 3.1 Let A be an m x n matrix. The orthogonal of the subspace 
S= {x E€ R” : Ax=0} 
is the subspace 


St = {x€ R” : x=A'u,u € R”}. 


Example 3.11 The line x, = —3x,, which passes through the origin, is a subspace of R?; 
see Figure 3.1. The equation of the line can be written as 


Ax = 0, where A= [31] and x= ie 
2 


Figure 3.1 A graphical illustration for the subspace in Example 3.11 and affine subspaces parallel 
to it. 


3.4 Convexity, Polyhedra, and Cones | 91 
All points on this line can be described by 
-1 
3 u, foru E€ R. 


So, the set {(—1, 3)" } is a basis for the given subspace and hence its dimension equals 1. Note 
that the vector (—1, 3)" is orthogonal to this subspace and every affine subspace parallel to 
it. To see this, note that by Proposition 3.1, we have 
È 
ER: [31] | =o} 
X2 


Å 
X 
{|$ | er: s+=0} - {|$ 
X2 X% 
Ji 3 
ER: x= 1 u,u E R 


x 
re Ap 
X% 2. |71 Xı 
ER’: =0>. 
X 3 Xz - 


In essence, matrices, subspaces, and bases are important concepts in linear algebra, and 
define the foundational framework for understanding and manipulating multidimensional 
spaces. 


ll 
Sy 
a 
CE 


3.4 Convexity, Polyhedra, and Cones 


This section introduces the definitions of convex functions, convex sets, convex hulls, poly- 
hedra, cones, and related notions. 


Definition 3.11 A function f : R” — R is called convex if for every x,y € R” and every 
A € [0,1], we have 


fax + (1 — Ay) < Af) + 1 — AF). 


To visually illustrate the above definition, see Figure 3.2. Examples of convex functions 
on their domains are x”, e*, and — log x. 


Definition 3.12 A set T C R” is said to be convex if Ax + (1 — Ayy E T for any x,y E T 
and å € [0,1]. 


See Figure 3.3, which shows convex and nonconvex sets in R?. We now introduce the 
convex hull of points in R”. 


Definition 3.13 A point x € R” is a convex combination of points of S C R” if there exist 
a finite set of points {x}‘_, in Sand 4 € R$ with Ð; 4; = 1 such that x = Ð; 4.x. The 
convex hull of S, denoted by conv(S), is the set all points that are convex combinations of 


points in S. 
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A 


flax+(1-A)y) 
i A f(x) + (1 ADO) 


> 
x Ax+(1-A)y y 
Figure 3.2 Illustration of the definition of a convex function on R. 
A convex set A nonconvex set 


Figure 3.3 A convex set versus a nonconvex set in R?. 
Figure 3.4 shows a one-dimensional convex hull and three two-dimensional convex hulls. 
Lemma 3.1 The convex hull of a finite number of vectors is a convex set. 


Proof: Let H be the convex hull of a finite number of vectors, say x®, x®, ...,x. Let also 
y= D4 X9 € H, z = Yi, Bx” € H, where o, p, > 0, and Xi a; = X; 2; = 1. Then, 
for any A € [0, 1], we have 


t 


t t 
Ay+(1-Az=A (Ze) =) (Zee) = Via; + (1 = Dp. 
i=1 i=1 


i=1 


Note that (Aa, +(1- 4p) =A Xi a, +0 -A DL, B= 44+ 1 - A) = 1. Hence, ay + 
pz € H. Thus, H is a convex set. The proof is complete. a 


A hyperplane is a subspace whose dimension is one less than that of its surrounding 
space. More formally, we have the following definition. 


3.4 Convexity, Polyhedra, and Cones 


Figure 3.4 Four convex hulls; three in R? and one in R. 


Definition 3.14 A hyperplane is the set of points that satisfy a linear equation. That is, if 
a € R” and b E R, then the subspace {x € R” : a'x = b} isa hyperplane in R”. 


A half-space is either of the two parts into which a hyperplane divides an affine space. 
More formally, we have the following definition. 


Definition 3.15 A half-space is the set of points that satisfy a linear inequality. That is, if 
a E€ R” and b E R, then the subspace 


H = {x€ R" : a'x>b} 


is a half-space in R”. 


A polyhedron is the intersection of a finite number of half-spaces. More formally, we have 
the following definition. 


Definition 3.16 A polyhedron is the set of points that satisfy a finite number of linear 
inequalities. That is, if A is an m x n matrix and b € R”, then the subspace 

P = {x ER": Ax > b} 
is a polyhedron in R”. 
Definition 3.17 A polyhedron P C R” is said to be bounded if there exists a positive con- 
stant k such that 

PC {xe R” : =k <x, <k for i = 1,2, ...,n}. 
Definition 3.18 A polytope is a bounded polyhedron. That is, if A is an m x n matrix, 


b € R” and kis a positive constant, then the subspace {x € [—-k,k]" : Ax > b} isa polytope 
in R”. 
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Lemma 3.2 A polyhedron is a convex set. 


Proof: Let P = {x € R” : Ax > 0} be a given polyhedron. Let also x,y € P and 4 € [0,1]. 
Then Ax > b and Ay > b. It follows that 
A(Ax + (1 — Ayy) = AAx + (1 — AAy > àb + (1 — A)b = b. 


Hence, ax + fy € P. Thus, P is a convex set. The proof is complete. E 
Definition 3.19 A set C C R” is called a cone if Ax € C for any x € C and å > 0. 
Lemma 3.3 The polyhedron P = {x € R” : Ax > 0} is a cone. 


Proof: Let x,y € P and a > 0. Then Ax > 0. It follows that A(ax) = «Ax > «0 = 0. Hence, 
ax € P. Thus, P is a cone. The proof is complete. a 


A convex cone is a cone that is also convex, that is, a cone that is closed under addition. 
So, a cone C is convex if C + C C C. The following combines Definitions 3.12 and 3.19. 


Definition 3.20 A set C C R” is a convex cone if ax + Py € C for any x,y E C and 
a,p> 0. 


The following is a corollary to Lemmas 3.2 and 3.3. It can be also proven directly using 
Definition 3.20. 


Corollary 3.4 The polyhedron P = {x € R” : Ax > 0} is a convex cone. 
We now introduce the conic hull of points in R”. 


Definition 3.21 A point x € R” is a conic combination of points of S C R” if there exist 
a finite set of points {x}'_, in S and 4 € R{ such thatx = Ð; 4x®. The conic hull of S, 
denoted by cone(S), is the set all points that are conic combinations of points in S. 


Let v4, Va, .-. , V, E R”. According to Definitions 3.13 and 3.21, we have 
conv(v,,...,v,) = {x ER” : x= Ji Avo 14 2 ads st}, 
A,20,i=1,...,t}. 


ii oe! 


cone(v;,...,0,) Ê {xER": x= Yi, Av 


Note that a convex combination differs from a conic combination in the sense that we do 
not require that S Ai = 1 in the latter. Figure 3.5 shows conic and convex hulls of points 
in R?. A cone is said to be regular if it is a closed, convex, pointed, solid cone. 


Definition 3.22 Let Y be a finite-dimensional Euclidean space over R with an inner 
product “(-,-).” The dual cone of a regular cone X C Y is defined as 


H*A(SEYV: (x,s)>0, VX EX}. 


A regular cone & is said to be self-dual if X = X *. 


3.5 Farkas’ Lemma and Its Variants 


e e 
e 
e e e 
e e 
e 
Points in R? Conic hull Convex hull 


Figure 3.5 The conic and convex hulls of some points in R?. 


Examples of self-dual cones include: 


e The cone of nonnegative orthant of R” (i.e., the polyhedron P = {x € R” : Ax > 0} with 
A equals the identity matrix); 

The second-order cone (see Lemma 11.3); 

The cone of real symmetric positive semidefinite matrices; 

The cone of complex Hermitian positive semidefinite matrices; 

The cone of quaternion Hermitian positive semidefinite matrices. 


Examples of non-self-dual cones include the pth-order cones (when p #2), the 
hyperbolic cones, the cone of copositive matrices, the doubly nonnegative cone, the power 
cone, and the exponential cone. 


3.5 Farkas’ Lemma and Its Variants 


Farkas’ lemma plays a central role in the development of the field of optimization, and more 
specifically, linear programming duality (see Chapter 10). In this section, we state and prove 
two versions of Farkas’ lemma. First, we give, without proof, the following result which will 
be used in the proof of Farkas’ lemma. 


Theorem 3.14 (Separating hyperplane) Let Q C R” be a closed, nonempty and con- 
vex set. Let b € R",b ¢ Q. Then there exist 0 4 v € R” and £ € R such that v'b > fp and 
v'q < f forall q € Q. 


For a proof of Theorem 3.14, see for example, Bertsimas and Tsitsiklis (1997). 


Theorem 3.15 (Farkas’ lemma (Version I)) LetA € R”" and b € R”. Then exactly 
one of the following two conditions holds for given A and b: 


(1) 3x € R” such that Ax = b and x > 0; 
(2) 3y € R” such that Ay > 0 and b'y < 0. 
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Proof: The proof consists of two steps. At first, we prove that we cannot have both (1) and 
(2) simultaneously, then we prove that if (1) does not hold then (2) does. 

Suppose, in the contrary, that we can have both (1) and (2) simultaneously. That is, there 
arex € R” andy € R” satisfying Ax = b and y'b < 0, with 0 < x and 0 < Ay. Then 


0 < (y"A) x =y" (4x) =y'b <0, 


giving the desired contradiction. Thus, we cannot have both (1) and (2) together. 

Now we prove that (2) holds if (1) does not hold. Let a,,a,,...,a,, be the columns of A. 
Then Ax = Dei xa. Note that the vector b satisfies Ax = b with x > Oifand onlyifb ¢ Q, 
where Q is the nonempty (as it contains 0), closed, convex set: 


n 
Qê cone(a,,...,a,,) = {dae 1x2 0,j=1,... n} ; 
j=1 
Therefore, if Condition (1) does not hold then b ¢ Q. The separating hyperplane theorem 
(Theorem 3.14) implies that there exists 0 4 v € R” and $ € R such that v'b> f} and 
v'q < f for all q E Q. In particular, because 0,x,a,,...,x,a, E Q, we have 0 = v'0< fp 
and v'(xja;) < $ for all j =1,2,...,n. Letting x; ——> oo, we have v'a, < B/x; — 0, 
which implies that v'a, < 0for all j =1,2,...,n. Picking y = —v, we get 


v'a 
v'a T T 
Aly=-Alv=-| .?|20, and b'y=-b'v < -f < 0. 
v'a, 
Thus Condition (2) holds. The proof is complete. a 


Theorem 3.16 (Farkas’ lemma (Version II)) Let A € R™®* and c € R”. Then exactly 
one of the following two conditions holds for given A and c: 


(1) 3y € R” such that Aly < c; 
(2) 3x € R” such that Ax = 0,c'x < 0 and x > 0. 


The following condition is equivalent to (2): 


(2') Ax € R” such that Ax = 0,c'x = —1 and x > 0. 


Proof: The proof consists of three steps or parts. At first, we prove that (2) and (2’) are equiv- 
alent. Then, we prove that we cannot have both (1) and (2) simultaneously. And finally, we 
prove that if (2) does not hold then (1) does. 

The proof of the first part is left as an exercise for the reader (see Exercise 3.11). Now, we 
prove that we cannot have both (1) and (2) simultaneously. Suppose, on the contrary, that 
we can have both (1) and (2) simultaneously. That is, there arey € R” and x € R” satisfying 
Aly < cand Ax = 0, with c'x < 0 and x > 0. Then 


0 = (Ax)'y = x'(A'y) < x'e < 0, 


giving the desired contradiction. Thus, we cannot have both (1) and (2) together. 


Exercises 


Finally, we prove if (2) is not true (and hence (2’) is not true as well), then (1) is true. 
Assume that (2’) does not hold, that is, there is no x satisfying Ax = 0, c'x = —1,and x > 0, 
or equivalently, satisfying Âx = b and x > 0, where 


a-a] ee-e] 


This means that Condition (1) of Farkas’ lemma (Version I) does not hold. It follows that 
Condition (2) of Farkas’ lemma (Version I) holds, that is, there is f satisfying A'y > 0and 
b'$ < 0. Let a be the last component of f, and p be the subvector of its remaining compo- 
nents. That is, f = (p', a)". Then b'f < 0 implies that a > 0. Also A'p > 0 implies that 


gea 


Note that (3.8) can be written as A'p + ac > Oor A'( =) < c. Thus, Condition (1) holds with 
y= =p. The proof is complete. a 


P| 50. (3.8) 
a 


In conclusion, Farkas’ lemma represents a fundamental theorem in linear programming 
and optimization. By considering both versions I and II of this lemma, we gain a more com- 
prehensive and versatile understanding of the relationships between feasible and optimal 
solutions in the world of linear programming. 


Exercises 


3.1 Choose the correct answer for each of the following multiple-choice questions/ 
items. 
(a) Which of the following sequences is bounded above? 


o { n2 \ (iii) {pn}. 


n+2 


a . 2” 
i [zaV i lar] 


(b) Which one of the following sequences diverges? 


3 2 
(i) { ay - \ (iii) {sme mn \ 


i l. 1 2 
(ii) { =In=}. i mtl 
non ) a n + 1000 f` 
(c) Which one of the following sequences converges to a number that is not 4/9? 
: 4n—-1 ra (2n + 1)? 
(i) { on \. (iii) { (n—1)2 \ 


cis 4n-1 
Gi) { 9n2 } (iv) {=}. 


9n2 
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3.2 


3.3 


3.4 


3.5 


3.6 


3.7 


3.8 


(d) The sum y 2" n equals 


(i) (1 + log, ,,n)n. (iii) (A + log n)n. 
(ii) n logn. Gv) nlogn. 
(e) Which of the following is a subspace of R?? 
(i) All vectors of the form (a, 0, 0)". 
(ii) All vectors of the form (a, 1, 1)". 
(iii) All vectors of the form (a,b,c)! where b =a+c+1. 
(iv) None of the above. 


(£) The value of a such that the vector (a, 7, —4)' is a linear combination of vectors 
(—2,2,1)' and (2, 1, —2)" is 
(i) 2. (ii) —2. (iii) 0. (iv) -1. 
(g) When the outer function is non-decreasing, the composition of two convex 
functions is 
(i) convex. (ii) nonconvex. 
(h) The intersection of two convex sets is 


(i) convex. (ii) nonconvex. 


Use induction and L’Hospital’s rule to show that, for each positive k, 


k 
iia = 0. 


n> n 
+ 60 
Find > ,-25 K”. 


Test the following series for convergence. If it converges, find its sum. 


` 1 
2 (2k+1X(2k-1) 


k=1 


Test the following series for convergence. 


so l oN 1 
U Esras roar 


Test the following series for convergence. If it converges, find its sum. 
a a 2\k wey Wang f2\? 
(i) dls) ‘ (iii) LD () : 
ay WI 5\e* wy (2)\* 
=i, 2 K($). 
(ii) 2 (3) (iv) 2 =) 


Prove that the inverse of a positive definite matrix is also positive definite. 


Find the orthogonal of the subspace H = {x € R? : x, = 2x,}. 


Notes and Sources 


3.9 Letfi.f..--.f, : R” > R be convex functions. Show that the function f defined by 
fœ) = max i, fi) is also convex. 


3.10 Prove that the intersection of a finite number of convex sets is convex. Argue if the 
intersection of an infinite number of convex sets is convex. 


3.11 Prove that Conditions (2) and (2’) in Theorem 3.16 (Farkas’ lemma (Version II)) are 
equivalent. 


Notes and Sources 


The histories of mathematical analysis and linear algebra date back to ancient civilizations, 
with roots in Babylon, ancient Greece, Arabia, and India. Looking at the analysis timeline, 
Eudoxus and Archimedes made significant contributions to the understanding of calculus- 
related concepts. However, the formalization and development of mathematical analysis 
as we know it today began in the 17th century with the work of European mathematicians 
like Isaac Newton. His book entitled “Philosophiae naturalis principia mathematica” (see 
Newton (1687)) marked a pivotal moment in the history of mathematical analysis. Over the 
subsequent centuries, mathematical analysis evolved and expanded, with contributions 
from other European mathematicians like Cauchy, Weierstrass, and Riemann; see for 
example Cauchy (1821). 

Looking at the algebra timeline, the term “algebra” itself is derived from the Arabic word 
“Al-Jabr,” introduced by Al-Khwarizmi in his work during the 9th century. Al-Khwarizmi’s 
book entitled “Kitab al-Mukhtasar fi Hisab al-Jabr wal-Muqabala” (see, for instance, 
Surhone et al. (2010)) laid the foundation for solving linear and quadratic equations, mark- 
ing a significant turning point in the development of algebra. Over the centuries, algebra 
evolved and expanded, with contributions from European mathematicians including 
Arthur Cayley. His work Cayley (1889) on matrices, in particular, laid the groundwork for 
the modern study of linear algebra. Mathematical analysis and linear algebra continued 
to flourish as essential branches of mathematics, with their modern forms being shaped in 
the 19th century. 

This chapter offered some fundamental concepts from mathematical analysis and linear 
algebra that serve as the basis for various fields, including combinatorics and optimiza- 
tion. It provided essential insights into foundational analytic and algebraic topics, such as 
sequences, series, matrices, subspaces, bases, polyhedra, and convex cones. Furthermore, 
we presented Farkas’ lemma, which is a pivotal result in linear programming. The chapter 
acts as an invaluable resource for individuals seeking an understanding of the analytic and 
algebraic structures that form the foundation of the disciplines under study in this book. 

As we conclude this chapter, it is worth noting that the cited references and others, such 
as Cusack and Santos (2021); Stewart (2015); Anton et al. (2021); Minton and Smith (2011), 
and Thomas et al. (2005), also serve as valuable sources of information pertaining to analytic 
structures covered in this chapter. Also, the cited references and others, such as Moh (2020); 
Neri (2016); Datta (2010); Meyer (2000); Bartl and Dubey (2017); Roos (2009); Jeyakumar 
(2009), and Bartl (2008) also serve as valuable sources of information pertaining to algebraic 
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and geometric structures covered in this chapter. We used and modified a code due to Stack- 
Exchange (2019) to create Figure 3.3. 
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Suppose that we want to visualize flight routes in North America. A natural representation 
is with a graph as follows. 


e Create a set of nodes (also called ver- 
tices), with each node representing a 
city. Each node is labeled by the city’s 
three-letter airport code. 

e Connect any two cities (nodes) which 
have a flight route between them 
with a line (called an edge). We can 


also label this edge with the mileage HNL LAX DFW (ua) 
of the route. ae, ae 


A way to think about graphs is: Some flight routes in North America. 


e Vertices (nodes) specify some entities we are interested in. 
e Edges specify the relationships between the entities (vertices). 


After we represent our data (flight) with a graph, we can answer a lot of interesting ques- 
tions such as: 


e Can we reach one city from another city? 
e What is the route with the minimum number of connections between two cities? 
e What is the minimum mileage route between two cities? 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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Many interesting questions can be answered efficiently, but for some, there is no hope of 
an efficient answer. This chapter is devoted to introducing graph concepts and terminology. 
Note that the graphs in this chapter are a completely different concept from the graphs used 
for plotting functions in calculus. 


4.1 Basic Graph Definitions 


In this section, we introduce some basic graph definitions. This includes directed and undi- 
rected graphs, simple and multigraphs, subgraphs, paths, cycles, connected components, 
trees, spanning trees, complete graphs, and bipartite graphs. 


Definition 4.1 A graph, G = (V, E), is a finite set of vertices, V, and a finite set of edges, 
E, where each edge (u, v) connects two vertices, u and v. 


Example 4.1 In the graph shown to the right, we have O) 
G=(V,E) where V = {u,v,w} is the set of the vertices and “ 

E = {(u, w), (u, v), (w, v)} is the set of edges. These vertices, namely () W) 
u, v, and w, form the fundamental building blocks of the graph. 
These edges, namely (u, w), (u, v), and (w, v), explicitly define the 
links between the respective vertices. m 


Graph of Example 4.1. 


Example 4.2 In the graph provided on the right, we have (b) 

G = (V, E) where V = {a,b,c,d,e} is the set of the vertices 

and E = {(a,c), (b,c), (c, e), (b, d), (d,e), (c,d)} is the set of aNd (a) 
edges. Overall, this representation of a graph, characterized a, 

by its vertices and edges, plays a pivotal role in visualizing ey 


the relationships and connections that exist within the given 


graphical representation. 
Graph of Example 4.2. 


E 
4.1.1 Directed and Undirected Graphs 


We classify the edges into two distinct categories: Directed edges and undirected edges. 
The directed edge has an origin vertex and a destination vertex, whereas for the undirected 
edge, there is no designated beginning or end to the edges. An easy way to think about a 
directed edge is a flight going from one vertex to another. If there is a flight going from 
San Francisco international airport (SFO) to Chicago O’Hare international airport (ORD), 
it does not necessarily mean that there is a flight going from ORD to SFO. On the other 
hand, an undirected edge can be thought of as a network of “friends.” For example, if 
Noah is a friend of Zaid, then this necessarily means that Zaid is also a friend of Noah. 
See Figure 4.1. 

This leads us to classify the graphs into two distinct categories: Directed graphs and undi- 
rected graphs. In directed graphs, all edges are directed. In undirected graphs, all edges are 
undirected (see Figure 4.1). Note that (u, v) and (v, u) are two different edges if the graph is 
directed, but they are the same edge if the graph is undirected. 


4.1 Basic Graph Definitions 


(a) 


(b) 


Figure 4.1 rah Ib) a forming a directed graph (a) and undirected edges forming an 
undirected graph (b 


beh. of Ti 


Not a graph. Not a graph. A graph if we look at it in 3D. 


Figure 4.2 Examples of graphs and non-graphs. 


In this chapter, we first study the undirected graphs, then we study the directed graphs. 
Applications of directed and undirected graphs include, but not limited to: 


e Transportation networks: City map, highway network, flight network, etc. 
e Computer networks: Local area network, internet, web, etc. 
e Linguistics, physics, chemistry, biology, etc. 


Two vertices u and v of a graph G are adjacent if there exists an edge (u, v) in G. In 
Figure 4.2, the left-hand and middle side pictures do not represent graphs, while the right- 
hand picture represents a graph because looking at it in three-dimensional space one finds 
that the vertices u and v are adjacent and the vertices w and x are adjacent. 


4.1.2 Simple and Multigraphs 


A self-loop is an edge that connects a vertex to itself. Multiedges (or parallel edges) are edges 
that have the same endpoints in undirected graphs, or the same origin and destination in 
directed graphs. 

In the graph shown to the right, there is a self-loop at vertex 
u and there are two parallel edges between the vertices u and v. 
If a graph does not have parallel edges and self-loops, then 
it is said to be simple. A multigraph can have multiple edges A non-simple graph. 
between the same two vertices and self-loops. 
In this chapter, we will deal almost exclusively with simple graphs. In other words, unless 
otherwise specified, when we say “graph” we mean “simple graph.” 


4.1.3 The Vertex Degree 


Let G = (V, E) be a graph. If u,v € V and (u, v) E E, we say that the edge (u, v) is incident 
to vertices u and v. The degree of a vertex v E€ V, denoted as deg(v), is the number of edges 
incident on v. A vertex whose degree is zero is called isolated. 
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For example, in the graph shown to the right, we have 


deg(i) = 1, deg(b) = deg(e) = 2, deg(c) = 3, deg(d) = 4, and (>) 

deg(a) = 0. So, a is an isolated vertex. The degree list of ae (@) 
an undirected graph is the nondecreasing sequence of its L 

vertex degrees. For instance, the degree list of the graph (i) O 


shown to the right is 0, 1, 2, 2, 3, 4. : 
A graph with a degree 


sequence 0, 1, 2, 2, 3, 4. 
4.1.4 Paths and Cycles 


A path P, from a vertex u to a vertex u’ is a graph written as a sequence (Vo, V4, Va» -..,U,) 
ofn + 1 vertices, such that u = vp and u’ = v,, and n edges (v,_,, v;) fori = 1,2,...,n. The 
length of a path is the number of its edges. So, the path P, has length n. A subpath ofa path 
is a contiguous subsequence of its vertices. A path is simple if all vertices in the path are 
distinct. Revisiting vertices and/or edges are allowed in nonsimple paths. Figure 4.3 shows 
simple and nonsimple paths. The simple path shown to the left has length 5 and has, for 
instance, the path (v, y, z, w) as a subpath. 

A cycle is a path on which the first vertex is equal to the last vertex and all edges are 
distinct. So, a cycle C,„ is a path P„ with vo = v,. In Figure 4.4, we show some cycles. A 
cycle is simple if all its vertices, except the first and the last one, are distinct. A graph with 
no simple cycles is called acyclic. In Figure 4.5, we show an acyclic graph, a simple cycle 
graph, and a nonsimple cycle graph. 


4.1.5 Subgraphs and Connected Components 


A graph H = (V’,E’) is called a subgraph of a graph G = (V, E) if V’C V and F' CE. A 
spanning subgraph of a graph G is a subgraph of G that contains all the vertices of G. That 
is, H =(V’, E’) is a spanning subgraph of a graph G = (V, E) if V’ = V and E’ C E. The 


Oge 
OD O-O 


A simple path: (u,v, y, z, w, x). A nonsimple path: (u,v, w, z, y, v,w, xX). 


Figure 4.3 Simple and nonsimple paths. 
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Figure 4.4 Cycle graphs. 
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An acyclic graph. A simple cycle C4: A nonsimple cycle: 


(v, u, Ww, x, Vv). v, u, y, W, X,Y, V). 


Figure 4.5 Acyclic and (simple and nonsimple) cycle graphs. 


WW ka 


The Petersen graph P. A subgraph of P. A subgraph of P. A span. subgraph of P. 


Figure 4.6 The Petersen graph P and some subgraphs of P. 


TAAR 


A connected graph. A connected graph. A disconnected graph. A disconnected graph. 


Figure 4.7 Connected and disconnected graphs. 


Petersen graph is well-known graph in graph theory.! Figure 4.6 shows three subgraphs of 
the Petersen graph, P, and only one of them is a spanning subgraph. 

A graph is called connected if every vertex is reachable from all other vertices, and dis- 
connected otherwise. That is, there is a path between every pair of distinct vertices of a 
connected graph. In Figure 4.7, we show four graphs; each one has 20 vertices; we see that 
the two left-hand side ones are connected, whereas the two right-hand side ones are dis- 
connected because the five external vertices are not reachable from any of other internal 
vertices. 

A connected component G’ of a graph G is a maximal connected subgraph of G. By max- 
imality we mean that there is no way to add into G’ any vertices and/or edges of G which 
are not currently in G’ in such a way that the resulting subgraph is connected. 


Example 4.3 The graph, say G, shown to the right in Figure 4.8 has nine vertices and 
seven edges. Note that the subgraph surrounded by the blue dashed lines is not a connected 
component of G. The reason for that is if we add a new vertex v, and a new edge (v3, v4), 
the resulting subgraph is still connected. 


1 The Petersen graph serves as a useful example and counterexample for many problems in graph theory. 
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Figure 4.8 A graph with three connected components. 


The graph G has three connected components, where each connected component is 
framed by a blue disk background shown to the left in Figure 4.8. Note that, for instance, 
if we add the vertex v, to the right- or left-hand side connected component, the resulting 
subgraph is not connected anymore. E 


Example 4.4 The graph shown in Figure 4.9 has 18 vertices, 12 edges, and 7 connected 
components. These components are framed by blue ellipsoid backgrounds. E 


The following remark can be directly proven on the basis of Definitions 2.7 and 2.8. 


Remark 4.1 The connected components of a graph define equivalence classes over its 
vertices under the “is reachable from” relation. 


From Theorem 2.1, the “is reachable from” relation defines a partition on the vertices, 
with two vertices being in the same partition (or the same equivalence class) if and only if 
they are in the same connected component. 


4.1.6 Trees and Spanning Trees 


One of the important classes of graphs is the trees. The importance of trees is evident from 
their applications in various areas, especially theoretical computer science and molecular 
evolution. A tree is a connected undirected graph with no cycles (i.e., connected, acyclic, 
and undirected graph). A forest is any undirected graph without cycles. Figure 4.10 shows 
a tree and a forest. Note that the connected components of a forest are trees. 

A spanning tree of a connected graph is a spanning subgraph that is a tree. Note that 
a spanning tree is not unique unless the graph is a tree. Figure 4.11 shows a graph and 
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Figure 4.9 A graph with seven connected components. 


4.1 Basic Graph Definitions | 111 


Figure 4.10 A tree graph (a) anda 


forest graph (b). (v1) (1) (v2) 


C) &) 3) &) C5 
C) 3) C9 C O W 


(a) (b) 


A graph G. A spanning tree of G. Another spanning tree of G. 


Figure 4.11 A graph with spanning trees. 


two of its spanning trees. Spanning trees have applications to the design of communication 
networks. 

A spanning forest ofa graph is a spanning subgraph that is a forest. Note that disconnected 
graphs have spanning forests and do not have spanning trees, and that only connected 
graphs have spanning trees. 


4.1.7 Complete and Bipartite Graphs 


A complete graph of n vertices, denoted as K,,, is a graph in which every pair of vertices is 
adjacent. In Figure 4.12, we show some complete graphs. An empty graph is a graph with 
no edges, that is, E = Ø. 

A bipartite graph is a graph in which its vertices can be partitioned into two sets such that, 
for every edge (u, v) in the graph, u is in one of the sets and v is in the other. A complete 
bipartite graph, denoted as K,, ,, is a bipartite graph in which each vertex in one of the sets is 
joined to each vertex in the other. Figure 4.13 shows bipartite and complete bipartite graphs. 

A planar graph is a graph that can be embedded in the plane. In other words, it can be 
drawn on the plane in such a way that its edges intersect only at their endpoints (i.e., no 
edges cross each other). See Figure 4.14. 


K3 


(or C3) K, Ks Ke 


Figure 4.12 Complete graphs. 
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A bipartite graph 
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Figure 4.13 Bipartite and complete bipartite graphs. 


(a) (b) 


Figure 4.14 A planar graph (a) and a nonplanar graph (b). 


Figure 4.15 A graph G (a) and its complement G (b). 
e 
(a) (b) 


The complement of a graph G = (V, E) is the graph G £ (V, E) on the same set of vertices 
as of G such that there will be an edge between two vertices u and v in V, if and only if there 
is no edge in between u and v in V. See Figure 4.15. 

A dense graph is a graph in which the number of edges is close to the maximal number 
of edges. The opposite, a graph with only a few edges, is a sparse graph. The distinction 
between sparse and dense graphs is rather vague and depends on the context. Formally 
speaking, a graph G = (V, E) is dense if |E] is close to |V?| (i.e., |E]  |V|?), and G is sparse 
if |E| is much less than |V?| (i.e., |E] ~ |V]). 

The matching (or assignment) problem is the problem of choosing an optimal assignment 
of a number of applicants to a number of jobs. We can model this with a bipartite graph. 
Here we assume that there are n, job applicants and n, job openings. Each applicant has 
a subset of the job openings s/he is interested in. Conversely, each job opening can only 
accept one applicant out of some subset of the applicants. In other words, there are certain 
allowed or “compatible” pairings of applicants to jobs. We can find an assignment of jobs 
for applicants in such a way all the pairings are allowed and as many applicants as possible 
get jobs. This can be modeled with a bipartite graph as follows: On the left side we have 
the applicants and on the right side we have the jobs. We draw an edge between applicant 
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A graph with three A graph with three 
edges and three edges and no bridge 
bridge edges. edges. 


Figure 4.16 Graphs with/without bridges. 


u and job v if they are compatible. The problem of finding a maximum matching in a given 
graph is an NP-hard problem in general (the class of NP-hard problems will be introduced 
briefly in Section 7.7), but if the graph is bipartite, the problem can be solved efficiently. 

A bridge or cut-edge is an edge of a graph whose deletion increases its number of con- 
nected components. In Figure 4.16, we show two graphs, one has three bridges while the 
other has no bridges. 

The first characterization of trees is that a graph is a tree if and only if it is connected 
and every edge is a bridge. Further characterizations of trees will be given in Section 4.2. 


4.2 isomorphism and Properties of Graphs 


In this section, we discuss graph isomorphism and present some graph properties. 


4.2.1 Graph isomorphism 


What does it mean for two graphs to be “the same?” We say that two graphs are isomorphic 
when the vertices of one can be relabeled to match the vertices of the other in a way that 
preserves adjacency. Formally, we have the following definition. 


Definition 4.2 We say that two graphs G and H are isomorphic if there is a bijection f : 
V(G) > V(H) such that any two vertices u and v of G are adjacent in G if and only if f(u) 
and f(v) are adjacent in H. 


See Figure 4.17 which shows an example of two isomorphic graphs. It is also not hard to see 
that the two graphs with solid vertices in Figure 4.18 are another example of an isomorphic 
pair of graphs. 

In fact, given two graphs, it is often really hard to tell if they are isomorphic, but it is 
usually easier to see if they are not isomorphic. We have the following remark which shows 
that isomorphic graphs have the same number of vertices and the same number of edges. 


Remark 4.2 If two graphs have different numbers of vertices or different numbers of 
edges, then they are not isomorphic. 


Figure 4.17 An isomorphism u G) Ow A bijection between 


between two graphs. V (G) and V (A): 


fw =u, fv) =v, 
so oo 


fQ =x, fe) =z. 
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Figure 4.18 A pair of isomorphic 
graphs. 


For example, the two connected graphs shown to the right 
have four vertices, but they are not isomorphic because they 


have different numbers of edges. 


Having the same number of vertices and the same number of edges is not a guarantee 
that two graphs will be isomorphic. We have the following remark which shows that 
isomorphic graphs have the same degree lists. 


Remark 4.3 _ If two graphs have different degree lists, then they are not isomorphic. 


As an example, the two connected graphs shown to the right 
have six vertices and six edges, but they are not isomorphic 
because they have different degree lists. 


Having the same number of vertices, the same number of edges, and the same degree lists 
is not a guarantee that two graphs will be isomorphic. We have the following remark which 
shows that isomorphic graphs have the same number of connected components. 


Remark 4.4 Iftwo graphs have different number of connected components (for instance, 
one graph is connected, and the other is not), then they are not isomorphic. 


As an example, the two graphs 


shown to the right have six vertices, ah a 7s 


six edges, and the degree list is the ° ° 


e 
same, but they are not isomorphic a, ws Ne? 


because one of them is connected 
while the other is not. 


C3 duplicated. The cycle Co. 
(Disconnected graph). (Connected graph). 


Finally, we also point out that having the same number of vertices, the same number 
of edges, the same degree lists, and the same number of connected components is not a 
guarantee that two graphs will be isomorphic. 

As an example, both the two connected graphs shown to the right have 
six vertices, eight edges, one connected component, and the degree list is <> 
the same, but they are not isomorphic. However, the graph on the top has a 
cycle of length 3, and the minimum length of any cycle in the graph on the 
bottom is 4. So these two graphs are not isomorphic. <> 
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4.2.2 Graph Properties 


In this part, we present some graph properties which are stated in the following theorems. 


Theorem 4.1 In an undirected graph G = (V, E), we have 
£ deg(v) = 2|El. 


veV 


Proof: Every edge (w, u) has two endpoints (w and u). The first endpoint w contributes 
exactly 1 to deg(u). The second endpoint u contributes exactly 1 to deg(w). Thus, each edge 
contributes exactly 2 to the sum on the left. This proves the theorem. a 


The following example is a direct application of Theorem 4.1. 


Example 4.5 In the graph, G = (V, E), shown to the right, we 


have (v2) 


4 
> deg(v) = Yi deg(v;) =3+ 243425 10 = 2|E|. 


bev a Ce hey 


This matches the result in Theorem 4.1. (v4) 
Throughout the rest of this section, for a given undirected 

graph G = (V, E), we let n = |V| and m = |E|. The following is Graph of Example 4.5. 

the bipartite version of Theorem 4.1. a 


Theorem 4.2 Ina bipartite graph G = (A UB, E), we have 
Dy deg(v) = £ deg(v) = m. 
vEA 


vEB 


Proof: Since A and B are partition sets of the vertex set, V, of the bipartite graph G (i.e., 
Au B= V and A NB = Ø), each vertex in V is either in A or B, but not both. Then, from 
Theorem 4.1, we have 


£ deg(v) = > deg(v) + £ deg(v) = 2m. 
veV vEA vEB 


Note that each edge has one endpoint in A and one endpoint in B. So, we have }, „<4 deg(v) = 
È eg deg(v). Thus, 


2) deg(v) = 25, deg(v) = 2m. 
vEA 


veB 


The result follows by dividing each term by 2. The proof is complete. a 


For example, in the bipartite graph shown on the left-hand side of Figure 4.13, we have 


> deg(v) = DS deg(v) = 8 = |E]. 


v in the first partite set v in the second partite set 


Theorem 4.3 In an undirected graph, we have m < n(n — 1)/2. 
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Proof: Form Theorem 4.1, we have 2m = }' .,, deg(v). Note that deg(v) < n — 1 because 
each vertex has degree at most n — 1. It follows that 


1 1 1x 1 
m= > 2, dego) < 5 iin 1) Pi 1) = Sn(n — 1). 


vEeV vEeV 


The proof is complete. a 


From Theorem 4.3, we can conclude the following corollary (for the definition of Big-O 
notation, see Definition 7.2). 


Corollary 4.1 In an undirected graph G = (V, E), we have |E| = O(V?).? 
The following theorem tells us when the inequality in Theorem 4.3 is actually an equality. 


Theorem 4.4 Ann-vertex complete graph has exactly n(n — 1)/2 edges. 


Proof: We prove that |E(K,,)| = n(n — 1)/2 by induction on n. The base case is trivial: K, 
has one vertex and hence no edges and n(n — 1)/2 = (1)(0)/2 = 0. Assume that the state- 
ment is true for n = 7, that is, |E(K,)| = 7(¢ — 1)/2, for some 7 € N. Now, we prove that 
|E(Kp41)| = (¢ + 1)2/2. Let H be a subgraph of K,,, created by removing any vertex v and 
all edges incident to v. Then H = K; and 


|E(K,4,)| = |E(K,)| + deg(v) = le -1)/2+ l = (C + DC/2. 


Thus, any n-vertex complete graph has exactly n(n — 1)/2 edges. The proofis complete. m 


For example, in the complete graph K, (see K; at the right-hand side of Figure 4.12), we 
have 6(6 — 1)/2 = 15 = |E]. 


Theorem 4.5 The number of odd-degree vertices is always even. 


Proof: By Theorem 4.1, the sum of all the degrees is twice the number of edges. Because 
the sum of the degrees is even and the sum of the degrees of vertices with even degrees 
is also even, the sum of the degrees of vertices with odd degrees must be even. Since the 
sum of the degrees of vertices with odd degrees is even, it must be an even number of those 
vertices. E 


For instance, in Example 4.5, the number of odd-degree vertices is two, which is even. 
We have also the following example. 


Example 4.6 Determine whether each of the following sequences is a degree sequence of 
a graph? Answer by “yes” or “no” and justify your answer. 


(a) (1,1,2,2, 2,3,3,3). (b) (0,0,1,1, 2, 2, 3, 3). (c) (0,0,1,1, 1,1, 6). 


2 To be consistent with the notations in the literature, throughout this book, we use O(V), O(V?), and 
O(V + E) to mean that O(|V|), O(|V|?) and O(|V| + |E|), respectively. 
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Solution 


(a) The answer is no because we have an odd number of odd-degree vertices. @———® 
(b) Yes. To justify this answer, a graph that represents this degree sequence @ fe) 
is given in Figure 4.19. WAI 
(c) No. The justification for this answer is that there is a vertex of degree 6. Figure 4.19 
So, the minimum number of vertices that this vertex must be adjacent to Graph of 
is 6. Therefore, all seven vertices must be connected. However, the degree Example 
sequence has two vertices of degree 0, contradicting this. 4.6(b). m 


The most important formula for studying planar graphs is Euler’s formula. 


Theorem 4.6 (Euler’s formula) Let G=(V,E) be a finite, connected, planar graph 
that is drawn in the plane without any edge intersections. Let also F be the set of faces 
(regions bounded by edges, including the outer, infinitely large region) of G, then 


IVI- |E| + |F| = 2. 


Proof: We prove the theorem by induction on m where m = |E|. If m = 0, then it consists 
of a single vertex with a single region surrounding it, and hence |V| — |E| + |F| =1 -0+ 
1 = 2. This proves the base case. 

For the inductive step, we assume that for some graph, the formula is true for any m = k 
for some k € N. To prove that the formula is true for m = k + 1, choose any edge e € E. Ife 
connects two vertices, then contracting it would reduce |V| and |E| by one. Otherwise, the 
edge e separates two faces, then removing it would reduce |F] and |E| by one. In either case, 
the result follows the inductive hypothesis. This proves the theorem. E 


As an illustration, in the graph shown to the right, we have 
|V|- |E| + |F| =6-84+4=2. 


We end this section with the following theorem which characterizes the tree graph. 


Theorem 4.7 Let G be an undirected graph on n vertices and m edges. The following 
statements are equivalent. 


(a) Gisa tree. 

(b) Any two vertices in G are connected by a unique simple path. 
(c) Gis connected but each edge is a bridge. 

(d) Gis connected and m = n — 1. 

(e) Gis acyclic and m = n — 1. 

(£) Gis acyclic but if any edge is added to G it creates a cycle. 


Proof: We need to show that (a) > (b) > (c) > (d) > (e) > (Ð > (a). 
(a) > (b): Given that every tree is connected, it guarantees the existence of at least one path 
connecting any pair of vertices in the graph G. To establish the uniqueness of this path, 


3 Euler’s formula was first proved by Leonhard Euler (1707-1783), a Swiss mathematician who made 
important and influential discoveries in many branches of mathematics. 
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we employ a proof by contradiction. Let us assume there are at least two distinct paths 
connecting a particular pair of vertices in G. The union of these paths would inevitably form 
a cycle within the graph, implying that G contains a cycle. However, this directly contradicts 
the established fact that G is a tree. Therefore, our initial assumption of multiple paths 
between vertices in G must be false, ultimately confirming the presence of a unique path 
between any two vertices in the tree. 

(b) > (c): Due to the existence of a distinct path connecting any pair of vertices in graph G, it 
can be unequivocally declared as connected. Now, consider an arbitrary edge (x, y) within G, 
denoting it as path P = (x,y) from vertex x to vertex y. Since this path is unique, its removal 
from G results in the absence of any alternative path between x and y, rendering G — (x,y) 
disconnected. This observation holds true for any edge (x, y) within G, demonstrating that 
the removal of any edge e from G results in a disconnected graph G — e. Thus, we conclude 
that while G itself is connected, the removal of any edge e from G leads to a disconnected 
graph. 

(c) > (d): By assumption, G is connected. So we need only to show that m = n — 1. We 
use proof by induction. It is clear that a connected graph with n = 1 or n = 2 vertices has 
n — 1 edges. Assume that every graph with fewer than n vertices satisfying (c) also satisfies 
(d). Let G be an n-vertex connected graph but G — e is disconnected for every edge e of 
G. Let e’ be any edge of G. Now, G — e’ is disconnected. Hence G — e’ has two connected 
components. Let G, and G, be the connected components of G. Let n; and m, be the number 
of vertices and edges in G,, for i = 1, 2. Now, each component satisfies (c) because n, < n for 
i = 1,2. By induction hypothesis, we have m; = n; — 1, for i = 1,2. So, m = m; +m, +1 = 
(n, —1)+(n, — 1) + 1 = n — 1. Thus, by mathematical induction, G has exactly n — 1 edges. 
(d) > (e): We have to show that every connected graph G with n vertices and n — 1 edges 
is acyclic. We use proof by induction. For n = 1,2, it is clear that all connected graphs with 
n vertices and n — 1 edges are acyclic. Assume that every connected graph with fewer than 
n vertices satisfying (d) is acyclic. Let G be an n-vertex connected graph with n — 1 edges. 
Because G is connected and has n — 1 edges, G has a vertex, say x, of degree 1. Let G’ = G — 
{x}. Then, G’ is connected and has n — 1 vertices and n — 2 edges. By induction hypothesis, 
G’ is acyclic. Because xis a 1-degree vertex in G, x cannot be in any cycle of G. Since G’ = G — 
{x} is acyclic, G must be acyclic. Thus, by mathematical induction, every n-vertex connected 
graph with n — 1 edges is acyclic. 

(e) > (f): Assume that G is acyclic and that m = n — 1. Let G,, 1 <i < k, be the connected 
components of G. Because G is acyclic, G; is acyclic fori = 1,2,...,k. Hence, eachG,,1 < i < 
k, is a tree. Let n; and m; be the number of vertices and edges in G; for each i = 1,2,...,k. As 
(a) implies (e), we have m = ba m; = YM, — 1) = n — k. Therefore, k = 1. This means 
that G is connected. Hence, G must be a tree. Since (a) implies (b), any two vertices in G 
are connected by a unique path. Thus, the addition of any new edge to G would inevitably 
create a cycle. 

(Ð > (a): The proof here is left to the reader as an exercise (see Exercise 4.5). a 


4.3 Eulerian and Hamiltonian Graphs 


This section introduces Eulerian and Hamiltonian paths and cycles. Eulerian graphs are 
motivated by the 18th-century problem about the seven bridges of Königsberg. 


4.3 Eulerian and Hamiltonian Graphs 


Figure 4.20 The graph of Königsberg bridge problem has four vertices a, b, c, and d (representing 
land masses) and seven edges (representing the bridges). 


4.3.1 Königsberg Bridge Problem 


Graph theory began in 1736 when Leonhard Euler solved the well-known Königsberg 
bridge problem. The city of Königsberg in Prussia (now Kaliningrad in Russia) is set on 
both sides of the Pregel river, which has seven bridges across it (see Figure 4.20). In 1735, 
citizens of the city were wondering if it is possible to: 


e Walk over all the bridges exactly once returning to where you started. 
e Walk over all the bridges exactly once ending at a different place than where you started. 


The negative resolution by Euler in 1736 laid the foundations of graph theory. 


4.3.2 Eulerian Paths and Cycles 


A graph is called Eulerian when it contains an Eulerian cycle. We have the following defi- 
nition. 


Definition 4.3 An Eulerian path in a graph is a path that visits every edge exactly once 
(allowing for revisiting vertices). An Eulerian cycle in a graph is an Eulerian path that starts 
and ends on the same vertex. 


Note that, from Definition 4.3, every Eulerian cycle is an Eulerian path. Figure 4.21 shows 
an Eulerian path and cycle. Note that, in the Eulerian path shown to the left in Figure 4.21, 
all vertices have even degrees. Note also that, in the Eulerian cycle shown to the right in 
Figure 4.21, there are exactly two vertices of odd degrees. The following two theorems, 
which are due to Euler, are regarded as an excellent characterization for Eulerian paths 
and cycles. 


Theorem 4.8 A connected undirected graph has an Eulerian cycle if and only if the 
degrees of all the vertices are even. 


Proof: An Eulerian cycle is a traversal of all the edges of a simple graph once and only 
once, staring at one vertex and ending at the same vertex. We can repeat vertices as many 
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An Eulerian path: (u,v, w, x, y, w). An Eulerian cycle: (u,v, w, x, y, w, u). 


Figure 4.21 An Eulerian path and cycle. 


times as we want, but you can never repeat an edge once it is traversed. We prove that if a 
connected undirected graph has an Eulerian cycle, then the degrees of all the vertices are 
even. Let G be a graph that has an Eulerian cycle. Every time we arrive at a vertex during 
our traversal of G, we enter through one edge and exit through another. Thus, there must 
be an even number of edges incident to every vertex. Therefore, every vertex of G has an 
even degree. This proves one direction of the theorem. The proof of the other direction is 
left to the reader as an exercise. a 


Theorem 4.9 A connected undirected graph has an Eulerian path that is not a cycle if 
and only if there are exactly two vertices of odd degrees. 


Proof: We prove that if a connected undirected graph has exactly two vertices of odd 
degrees, then it has an Eulerian path. Let G be a graph with exactly two vertices, say u 
and v, of odd degrees. Adding the edge (u, v) to G forms a graph having all even degrees. 
By Theorem 4.8, the resulting graph contains an Eulerian cycle. This cycle uses the edge 
(u, v). Thus, deleting the edge (u, v) from the resulting graph yields an Eulerian path in the 
graph G. This proves one direction of the theorem. The proof of the other direction is left 
as an exercise. a 


To illustrate the usefulness Theorems 4.8 and 4.9, we present some examples. 


Example 4.7 Determine if the following graphs have an Eulerian cycle, Eulerian path, or 
neither? If there is one, give it. Otherwise, justify why there is none. 


v Ww v w 
S 

u x oan © u X 
(b) 


(a) (c) 


Solution 

(a) Note that all vertices in the graph are of even degrees. From Theorem 4.8, the graph has 

an Eulerian cycle. Also, from Theorem 4.9, the graph does not have an Eulerian path. 

The Eulerian cycle is (u, v, w, x, u). 

(b) Note that all vertices in the graph are of even degrees. From Theorem 4.8, the graph has 

an Eulerian cycle. Also, from Theorem 4.9, the graph does not have an Eulerian path. 

The Eulerian cycle is (y, v, W, Z, X, U, U, X, W, U, y). 

(c) Note that all vertices in the graph are of odd degrees. From Theorem 4.8, the graph 
does not have an Eulerian cycle. Also, from Theorem 4.9, the graph does not have an 
Eulerian path. a 
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Example 4.8 Determine if K,, has an Eulerian cycle, Eulerian path, or neither. Justify 
your answer. 


Solution 

The complete bipartite graph K,, has 12 vertices: 7 vertices of 
degree 5, and 5 vertices of degree 7. By Theorem 4.8, because not 
all vertices of K; are of even degrees, the graph K;, does not 
have an Eulerian cycle. In addition, by Theorem 4.8, because 
K; does not have exactly two vertices of odd degrees, the graph 
K; does not have an Eulerian path. 


Finally, revisiting the Königsberg bridge problem, note that the degrees of all four ver- 
tices of the graph shown in Figure 4.20 are odd. So, applying Theorems 4.8 and 4.9 to the 
Königsberg bridge graph, we understand now why Euler was led to a negative resolution of 
the problem. 


4.3.3 Hamiltonian Paths and Cycles 


A graph is called Hamiltonian when it contains a Hamiltonian cycle. We have the following 
definition. 


Definition 4.4 A Hamiltonian path in a graph is a path that visits every vertex in the 
graph exactly once. A Hamiltonian cycle in a graph is a Hamiltonian path that starts and 
ends on the same vertex. 


See Figure 4.22, which shows a Hamiltonian path and Hamiltonian cycles. 

Note that if a graph has a Hamiltonian cycle, then it automatically has a Hamiltonian 
path by deleting any edge of the cycle. 

The following theorem is due to Dirac. The proof of this theorem is beyond the scope 
of this book and is omitted. The proof can be found in any standard textbook on graph 
theory. 


Theorem 4.10 If each vertex of a connected graph with n > 3 vertices is adjacent to at 
least n/2 vertices, then the graph has a Hamiltonian cycle (hence, also has a Hamiltonian 
path). 


A Hamiltonian path. A Hamiltonian cycle. A Hamiltonian cycle. 


Figure 4.22 A graph with a Hamiltonian path and graphs with Hamiltonian cycles. 
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Figure 4.23 Some bipartite graphs. Among them, only K,, satisfies the hypothesis of Dirac’s 
theorem, hence it is Hamiltonian. 


Note that the converse of Theorem 4.10 is not necessarily true, that is, ifa connected graph 
with n > 3 vertices has a Hamiltonian cycle, then it is not necessarily that each vertex of 
the graph is adjacent to at least n/2 vertices. The graph shown on the right-hand side of 
Figure 4.22 is an example, and, indeed, many other graphs can also be taken to verify that 
the converse of Theorem 4.10 is not true in general. For example, any circle C,,, where n > 5, 
can be considered. 

Note also that, if n is odd in Theorem 4.10, we take the 
ceiling of n/2. In the graph shown to the right, we give 
a 5-vertex graph, in which the degree of each vertex 
is greater than or equal [5/2] =2, but the graph is 
non-Hamiltonian. (Why?) A non-Hamiltonian graph. 


Figure 4.23 shows some bipartite graphs. Among them, only the complete bipartite 
graph K, satisfies the condition that each vertex has a degree at least half the number of 
vertices. Based on Theorem 4.10, K,, has a Hamiltonian cycle. It is also not hard to see that 
the complete bipartite graph K; , has a Hamiltonian path but does not have a Hamiltonian 
cycle. 

Generally, from Theorem 4.10, any complete bipartite graph K, n» 
Hamiltonian cycle (hence, has also a Hamiltonian path). Additionally, the complete bipar- 
tite graphs K, n41 and K,,,;,,, where n > 1, have Hamiltonian paths but have no Hamiltonian 
cycles. Finally, from Theorem 4.10, any complete graph K,,, where n > 2, has a Hamiltonian 


cycle (hence, also has a Hamiltonian path). 


where n > 2, has a 


4.4 Graph Coloring 


We start with the following example inspired by Hussin et al. (2011). It introduces graph 
coloring and motivates the reader to some of its applications. 


Example 4.9 Every semester, the registrar ensures that no student faces the inconve- 
nience of having to take two exams simultaneously. To address this challenge, we employ 
a graph modeling approach, creating a graph denoted as G = (V, E). Here, the vertex set 
V directly corresponds to all classes, assigning one vertex per class. The set E is defined 
based on student enrollments: an edge connects vertices u and v if there exists at least 
one student who is enrolled in both class u and class v. Our objective is to allocate each 
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Figure 4.24 Scheduling final 
exams for 12 classes by coloring a 
graph of 12 vertices using four 
colors (a) and coloring the same 
graph using only two colors (b). 


vertex to a specific time slot while ensuring that vertices connected by an edge are assigned 
distinct slots. 

The most straightforward approach is to assign each final exam its dedicated time slot, 
resulting in n time slots if there are n classes. However, this can lead to an overly long final 
exam period. Therefore, we are looking for a solution that not only satisfies the scheduling 
requirements but also minimizes the total number of time slots. 

One way to conceptualize the concept of “assigning a time slot” is to view each time slot 
as a unique color. In essence, given graph G, our task is to color each vertex in such a way 
that connected vertices do not share the same color. 

In Figure 4.24, we illustrate this process by coloring a graph with 12 vertices, showcasing 
how to schedule final exams for these 12 classes. Each class corresponds to a vertex labeled 
as c for i = 1,2, ...,12. It is assumed that every class c, contains exactly three students, 
each enrolled in three different classes aside from c,. Figure 4.24 (a) demonstrates a col- 
oring approach utilizing four distinct colors, while the right side employs only two colors, 
presenting alternative scheduling possibilities. a 


Now we are ready to introduce the properly colored graphs. Formally, we have the 
following definition. 


Definition 4.5 A proper coloring is an assignment of colors to the vertices of a graph so 
that no two adjacent vertices have the same color. 


Figure 4.25 shows a properly colored graph and an improperly colored graph. Note that the 
two colorings used for the 12-vertex graph shown in Figure 4.24 are both proper. 


Definition 4.6 A (proper) k-coloring of a graph is a proper coloring using at most k colors. 
A graph that has a k-coloring is said to be k-colorable. 


oO © O oO @ ® 

~] | [Sa 

O @ O oO @ O 
(a) (b) 


Figure 4.25 Proper graph coloring (a) versus improper graph coloring (b). 
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Clearly, every graph G = (V, E) is |V|-colorable because we can assign a different color to 
each vertex. We are usually interested in the minimum number of colors we can get away 
with and still properly color a graph. We have the following theorems. 


Theorem 4.11 A graph is 2-colorable if and only if it is bipartite. 


Proof: Consider a graph G that is 2-colorable, meaning we can assign either red or blue to 
each vertex in such a way that no edge connects two vertices of the same color. We denote 
the subset of red-colored vertices as A and the subset of blue-colored vertices as B. Since all 
vertices within set A are red, there are no edges connecting them, and the same holds for 
set B. Consequently, every edge in the graph connects one endpoint in set A and the other 
in set B, which defines G as bipartite. 

Conversely, let us assume that G is a bipartite graph, implying we can partition its vertices 
into two distinct subsets, A and B, where every edge connects a vertex in A to one in B. By 
coloring every vertex in A as red and every vertex in B as blue, we get a proper 2-coloring of 
G. Therefore, G is indeed 2-colorable. E 


Theorem 4.12 The complete graph K, is not (n — 1)-colorable. 


Proof: Consider any vertex coloring of the complete graph K, that employs a maximum of 
n — 1 colors. Given that there are precisely n vertices, it follows that there must be at least 
one pair of vertices, say u and v, which are assigned the same color. However, (u, v) forms 
an edge within K,,. This particular edge possesses two endpoints sharing the identical color. 
Consequently, the coloring is improper, leading to the conclusion that K,, cannot be colored 
with just n — 1 colors. E 


Definition 4.7 The chromatic number ofa graph G is denoted by y (G) and is defined to be 
the minimum positive integer k such that G is k-colorable. A graph G is called k-chromatic 
if y(G) =k. 


From Definition 4.7, a graph is called k-chromatic if it is k-colorable, but not (k — 1)- 
colorable. For example, by Theorem 4.12, the complete graph K,„ is n-chromatic (hence 
(Ka) = n). Note that, from Theorem 4.11, a graph G is bipartite iff y(G) < 2, and the equal- 
ity holds iff G has at least one edge. In particular, the complete bipartite graph is 2-chromatic 
(hence 7(K,,,) = 2). See Figure 4.23. We also have the following theorem. 


Theorem 4.13 If H is a subgraph ofa graph G, then y(H) < x(G). 


Proof: Let G be a graph that has H as a subgraph. Let also y(G) = k, then G is k-colorable, 
hence so is H. This implies that y(H) < k = y(G). E 


Finding the chromatic number of a graph is an optimization problem. In general, when 
x(G) > 3, computing the chromatic number of G is an NP-hard problem, which belongs to 
a class of problems that will be introduced briefly in Section 7.7. 


4.5 Directed Graphs 
4.5 Directed Graphs 


All graphs that we have looked at so far are undirected graphs. Their edges are also said to be 
undirected. Sometimes it is necessary to associate directions with edges and get a directed 
graph instead. 

Before formally introducing the definition of a directed graph, it is worth mentioning that 
we can use relations in this definition. The representation of a relation # on a set A can be 
introduced with a directed graph as follows: A collection of vertices, one for each element 
in A, and directed edges, where an edge exists from vertex u to vertex v if and only ifu Z v. 
It is also standard to treat a directed graph as we define it below. 


Definition 4.8 A directed graph, or digraph, is a graph whose edges have a defined direc- 
tion, usually edges are represented as ordered pairs where the ordered pair (u, v) indicates 
that there is a directed edge from vertex u to vertex v. 


A slightly different set of definitions are used for directed graphs. A simple digraph has 
no loops and no multiple edges. See Figure 4.26. All directed graphs that we consider in the 
book are simple. 


4.5.1 Vertex In-Degree and Out-Degree 


If e = (u, v) is a directed edge in a digraph, then e is called incident from u and incident 
to v. The in-degree of a vertex v, denoted as deg,,,(v), is the number of edges incident to v 
(i.e., those entering the vertex v). The out-degree of a vertex v, denoted as deg,,,,(v), is the 
number of edges incident from v (i.e., those leaving the vertex v). For example, in the graph 
shown on the left-hand side of Figure 4.26, we have deg,,,(v) = deg,,,,(u) = 2 and deg, ,(v) = 
deg;,(u) = degoul wW) = deg;,(w) = 1. 

A vertex v is called balanced ifits in-degree is equal to its out-degree. A directed graph is 
balanced if every vertex is balanced. In Figure 4.27, we show two balanced directed graphs 


Figure 4.26 A simple digraph (a) versus a nonsimple 
digraph (b). 1 ? 
(b) 


(a) 


Figure 4.27 Balanced digraphs. 
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in which each vertex has two incoming and two outgoing edges, except the nonsolid vertex 
which has six incoming and six outgoing edges. 
The following is the directed version of Theorem 4.1. 


Theorem 4.14 Ina directed graph G = (V, E), we have 
F degno) = F degour(v) = |E|. 


vEeV veV 


Proof: Every directed edge (w, u) has two endpoints (w and u). The first endpoint w con- 
tributes exactly 1 to deg,,,,(w). The second endpoint u contributes exactly 1 to deg; (u). Thus, 
each directed edge contributes exactly 1 to exactly one of the sums on the left. The proof is 
complete. E 


The simple calculations in the following example match the result in Theorem 4.14. 


Example 4.10 In the digraph, G = (V, E), shown to the 


right, we have (v2) 
4 
F degno) = F degn (o) =3+0+1+1=5= |F| C (v3) 


veV i=1 
and (v4) 
4 
F degou) = F degout) =0+2+2+1=5= |El. Digraph of Example 4.10. 
veV i=1 a 


4.5.2 Directed Paths, Cycles, and Trees 


A directed path, or dipath, in a directed graph is a sequence of edges joining a sequence of 
distinct vertices where the edges are all directed in the same direction. See Figure 4.28. 

A directed cycle, or dicycle, is a directed path (with at least one edge) whose first and last 
vertices are the same. A directed tree, or ditree, is a directed graph whose underlying graph 
is a tree. A directed forest is a family of disjoint directed trees. See Figure 4.29. 

A semicycle is a directed cycle in which some of its edges have been reversed. In other 
words, a semicycle constitutes an (undirected) cycle when neglecting its directionality. A 
cycle in a digraph is either a directed cycle or a semicycle. For example, the circle given by 
the sequence (v, w, x, y, v) in the digraph shown in Figure 4.30 (c) is a directed cycle, whereas 
that given by the same sequence in the digraph shown in Figure 4.30 (b) is a semicircle. 

After introducing the weakly connectedness in this section, we can see that a directed 
tree can now be redefined to be a weakly connected digraph that has no cycles. 


C) © C) © Figure 4.28 A valid directed path 
from u to z (a) versus a “nonvalid 
Ce) O © Ce) S o path” from u to z (b). 
(a) (b) 
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(A 


(a) (b) (c) 


Figure 4.29 A directed cycle (a), a directed tree (b), and a directed forest (c). 
OOOO OD OF 
OOOO POTO EOE 


(a) 
Figure 4.30 A directed tree (a), a DAG (b), and a non-DAG (c). 


We also have the following definition. 
Definition 4.9 A directed acyclic graph (DAG) is a digraph that has no directed cycles. 


Figure 4.30 shows a directed tree, a DAG, and a non-DAG. In Chapter 9, we will see that 
a DAG can be redefined to be a digraph that has the so-called topological ordering. 


4.5.3 Connectedness 


A directed graph is called weakly connected if replacing all of its directed edges with undi- 
rected edges produces a connected (undirected) graph. A directed graph is called strongly 
connected if every two vertices are reachable from each other. In other words, a digraph 
strongly connected if there is a path in each direction between each pair of vertices of the 
digraph. See Figure 4.31. 

A strongly connected component G’ of a digraph G is a maximal strongly connected 
directed subgraph of G. By maximality, we mean that there is no way to add into G’ any 
vertices and/or directed edges of G which are not currently in G’ in such a way that the 
resulting directed subgraph is strongly connected. 


The vertex y is not D © In the graph shown G © 


reachable from the on the right, every 


vertex w, but the two vertices are 

underlying reachable from each 

undirected graph is other. 

connected. G) (w) G) (wv) 
(a) (b) 


Figure 4.31 A weakly connected digraph (a) versus a strongly connected digraph (b). 
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Figure 4.32 A Figure 4.33 A digraph with five 
digraph with strongly connected components. 
three strongly 

connected 

components. 


The digraph shown in Figure 4.32, for example, has three strongly connected components 
which are surrounded by blue dashed polygons. For another example, the digraph shown 
in Figure 4.33 has five strongly connected components which are surrounded by blue 
dashed polygons. Note that a strongly connected digraph has only one strongly connected 
component. 


Remark 4.5 The strongly connected components of a digraph define equivalence classes 
over its vertices under the “is mutually reachable from” relation. 


Remark 4.5 can be directly proven on the basis of Definitions 2.7 and 2.8. 


Exercises 


4.1 Choose the correct answer for each of the following multiple-choice questions/ 


items. 

(a) Ifasimple graph has three vertices, then the number of vertices with odd degrees 
can only be: 
(i) 0. (ii) 1. (iii) 2. (iv) 0 or 2. 

(b) What is the number of edges present in a graph having 244 degrees total? 
(i) 61. (ii) 122. (iii) 244. (iv) 488. 


(c) Let G = (V, E) be an undirected graph. The following statements are equivalent, 
except for one that is not. Which one? 
(i) Any two vertices in G are connected by a simple path. 
(ii) Gis connected but each edge is a bridge. 
(iii) Gis connected and |V|? = |E|? + 2|E| + 1. 
(iv) Gis acyclic but if any edge is added to G it creates a cycle. 


(d) Let G be a simple connected graph with n vertices, where n is an odd number 
greater than 2. Which one of the following statements is the contrapositive of 


Dirac’s theorem for graphs? 
(i) If every vertex of G is adjacent to at least (n + 1)/2 vertices, then G has a 


Hamiltonian cycle. 
(ii) If G has no Hamiltonian cycle, then every vertex of G is adjacent to at most 
(n + 1)/2 vertices. 


4.2 


4.3 


4.4 


4.5 
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(iii) If G has no Eulerian cycle, then each vertex of G is adjacent to less than 
(n + 1)/2 vertices. 
(iv) Every vertex of G has degree at most n/2 if G has no Hamiltonian cycle. 
(e) Let p and q be even numbers greater than two. Which one of the following state- 
ments is false? 
(i) The complete bipartite graph K, , has an Eulerian cycle. 
(ii) The complete bipartite graph K, , has an Eulerian path. 
(iii) The complete graph K, has no Eulerian cycle. 
(iv) The complete bipartite graph K, , has an Eulerian cycle. 


(f) Let p be an integer number greater than one. Which one of the following graphs 
has no Hamiltonian cycle? 
(i) The complete graph K,. (iii) The complete bipartite K, p41- 
(ii) The complete bipartite K, p. (iv) The cycle C}. 
(g) The chromatic number of an even cycle (i.e., a cycle with even length) equals: 
(i) 2. (ii) 3. (iii) 2 or 3. (iv) its length. 
(h) Let G=(V,E) be a digraph, where V = {a,b,c,d,e,f,g,h,i} and E = {(a, d), 
grap E 
(d, a), (b,c), (c, f), f, b), (g, h), h, 8), (b, e), (e, d), (e, i), (h, e)}. What is the number 
of strongly connected components in G? 
(i) 3. Gi) 4. (iii) 5. (iv) 6. 
(i) Which one of the following statements is true? 
(i) There are two isomorphic graphs with different degree sequences. 
(ii) There are no two nonisomorphic graphs with the same degree sequence. 
(iii) There are no two isomorphic graphs with the same degree sequences. 
(iv) There are two nonisomorphic graphs with the same degree sequence. 


Find a spanning tree for the following graph so that no vertex has a degree of 4. 


> 


Are the graphs G, and G, shown below isomorphic? Why or why not? 


G] G2 
Determine whether each of the following sequences is a degree sequence of a graph? 
Answer by “yes” or “no” and justify your answer. 


(a) (1, 2, 3, 3, 4, 5, 5). (b) (0,0,1,1,1,1,4,4). (c) (0,2, 2, 2, 2, 4, 4, 4). 


Prove that the direction “(f) > (a)” in Theorem 4.7. 
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4.6 Give an undirected graph with five vertices, all of which must be at least degree 3. It 
must have an Eulerian cycle and a Hamiltonian cycle. Can you give another undi- 
rected graph? 


4.7 Does the bipartite graph K; , have a Hamiltonian cycle or path? Justify your answer. 
4.8 Let Gbeak-colorable simple graph. Provide a tight upper bound‘ on the maximum 
number of colors that one could need to properly color the graph if 
(a) one adds an edge between two vertices in G. 


(b) one removes an edge from G. 


4.9 Isthe statement that “Any graph with a vertex of degree d is d + 1-colorable” true or 
false? If it is false, explain why. 


4.10 Find the strongly connected components of the digraph shown in Figure 4.34. 


(v2) v7 Figure 4.34 Digraph of Exercise 4.10. 
is 
(v1)< 3} >| V6 > V8 


Notes and Sources 


Graph theory has its roots in the 18th century. The Swiss mathematician Leonhard Euler 
is often credited with pioneering graph theory through his work on the “Seven Bridges of 
Königsberg” problem in 1735; see Euler (1736). Euler’s solution to this problem laid the 
foundation for the study of graphs and led to the development of graph theory as a distinct 
mathematical discipline. The theory continued to evolve in the 19th and 20th centuries, 
with contributions from mathematicians like Gustav Kirchhoff and Arthur Cayley, who 
made significant advancements in the study of graphs and their applications in various 
fields, see for instance Cayley (1889). 

This chapter served as a comprehensive introduction to essential concepts in graph 
theory, providing a foundation for both theoretical understanding and practical problem- 
solving in diverse fields. It covered key topics such as graph properties, graph coloring, and 
directed graphs. Readers also gained an understanding of how graphs could be analyzed 
and compared for structural similarity through isomorphism. We studied special graphs, 
including Eulerian and Hamiltonian graphs, and explored the art of graph coloring with 
applications. 


4 By a tight upper bound, we look for an “interesting answer.” For example, the upper bound |V] is not an 
interesting answer in terms of credit. 
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As we conclude this chapter, it is worth noting that the cited references and others, such 
as Rosen (2002); Cusack and Santos (2021); Mott et al. (1986); Diestel (2012); Harris et al. 
(2008); Bondy and Murty (2008); Bollobas (2002); Godsil and Royle (2001); Erciyes (2021); 
Rahman (2017); Kagaris and Tragoudas (2008), and Gabow et al. (2003), also serve as valu- 
able sources of information pertaining to the subject matter covered in this chapter. The 
code that created Figure 4.13 is due to StackExchange (2011). The code that created the left- 
hand side picture in Figure 4.20 is due to StackExchange (2014). We modified a code due 
to StackExchange (2015) to create Figure 4.24. We modified a code due to StackExchange 
(2020) to create Figure 4.27. 
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Within this chapter, we acquire knowledge of various methods for solving recurrences. A 
formal definition of a recurrence looks like: 


Definition 5.1 A recurrence is an equation or inequality that describes a function in 
terms of its value on smaller inputs. 


There are five efficient methods for solving recurrences, namely the guess-and-confirm 
method, the iteration method, the recursion-tree method, the generating functions method, 
and the master method. In this chapter, we present the first four methods. The fifth method 
(master method) is beyond the scope of this book. However, there are a number of good 
references for solving recurrences using the master method, see for example (Cormen et al., 
2001, Section 4.5). 


5.1 Guess-and-Confirm 


In this section, we learn what we mean by solving recurrences and then present the guess- 
and-confirm method for solving them. 
Asa matter of example, it can be shown (see Exercise 5.2 (b)) that the following recurrence 


TQ) =1, Tn) =3T(n-1)+4, n=2,3,..., (5.1) 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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describes the function 
T(n) = 3"-2, n=1,2,.... (5.2) 


An explicit formula (also called a closed formula) is said to be a solution of a recurrence 
relation if its terms satisfy the recurrence relation. For example, the formula (5.2) is the 
solution of the recurrence relation (5.1). The initial conditions for a recurrence relation 
specify the terms that precede the first term where the recurrence relation takes effect. For 
example, T(1) = 1 is the initial condition for the recurrence relation (5.1). 

The following are some instances of recurrence formulas. 


e Arecurrence formula that loops through the input to eliminate one item. For example: 
T(n) = T(n—1)+1, T(n) = Tn-1) +h, etc. 

e Arecurrence formula that halves the input. For example: 
T(n) = T(n/2) +1, T(n) = T(n/2) +n, etc. 

e Arecurrence formula that splits the input into two halves. For example: 
T(n) = 2T(n/2) + 1, etc. 


Each of Sections 5.1-5.4 describes a method for solving recurrence. 

As we mentioned earlier, the technique under discussion in this section is the guess-and- 
confirm method. This method follows a systematic approach in which we initially employ 
repeated substitutions to make an educated guess or hypothesis regarding the explicit form 
ofa recurrence formula. Once we have this proposed expression, we proceed to utilize math- 
ematical induction, a fundamental proof technique of Section 2.1, to rigorously establish 
and validate the accuracy of our conjecture. Due to this dual-step process, the guess-and- 
confirm method is also recognized as the substitution/induction method, as it distinctly 
involves both of these fundamental components. 

To illustrate the practical application of this method, we will provide examples that show- 
case its effectiveness and how it is systematically employed to derive and validate recurrence 
formulas. Through these examples, readers will gain a deeper understanding of the guess- 
and-confirm method and its value as a tool in solving recurrence relations. 


Example 5.1 Use the guess-and-confirm method to solve the following recurrences. 


C1, ifn=1; 
T(n—1)+c,, ifn>1. Here, c, and c, are constants. 


(a) Tin)! = { 


0 ifn = 1; 
2 = > EJ 
(o) T= { T(n/2)+1, ifn> 1, and n isa power of 2. 


1 This recurrence formula is the one that arises from the factorial function algorithm (Algorithm 7.28). 
2 This recurrence formula is the one that arises from the binary search algorithm (Algorithm 8.6). 
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Solution 
(a) Using repeated substitutions, we have 


TQ) = cĉ, 
T(2) = TA) +c = c +c, 

T(3) = T2) +c, = & + 2c, 
T(4) = TB) +c, = c +30, 


T(n) = T(n- 1)+c, = & +(n- 1)c,, forall n > 1. 
We prove the nth-term guess by mathematical induction. The base case is trivial: 
T(1) =c,. For the inductive hypothesis: we assume that T(k) =c, + (k — 1)c,, for 
k < n. We now use the inductive hypothesis to prove the statement for k + 1. Note that 
T(k+1)=¢,+ T(k) =c,+¢,+(k—-1)c, = c + ke,, 
which completes the inductive step. Thus, T(n) = c, + (n — 1)c, for n > 1. 
(b) Using repeated substitutions, we have 
TQ) = 0 = log1, 
T2) = T2/2)+1=TA)+1=0+1=1 = log2, 
T(4) = T(4/2)+1 = T(2)+1 =1+1=2 =log4, 
T(8) = T(8/2) +1 =T(4)+1=2+1=23 = log8, 


T(n) = logn, where n > 1 and n is a power of 2. 


We prove the nth-term guess by mathematical induction. The base case is trivial: 
T(1) = 0 = log 1. For the inductive hypothesis: we assume that T(k) = logk, for k < n. 
We now use the inductive hypothesis to prove the statement for n. Note that 


T(n) = T(n/2) + 1 = log(n/2) + 1 = logn — log2 +1 = logn. 
This completes the inductive step and hence completes the proof. E 


Example 5.2 Use the guess-and-confirm method to determine a good upper bound on 
the following recurrence. 


T(n) = 1, ifn = 0,1; 
~ | Tn-1)+T(n-2)4+1, ifn>1. 


Solution 

Using repeated substitutions, we have 
TO) =1<2°, 
TQ) =1<2}, 


T(2) = T()+T(0)+1=3<2?, 
T3) = T2)+T0)4+1=5 <2}, 
T(4) = T3)+7T(2)4+1=9 < 2%, 
T(5) = T4)+73)4+1=15 < 25, 


T(n) = T(n-—1)+T(n-2)+1< 2", foralln>1. 
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We prove the nth-term guess, which is T(n) < 2”, by induction on n. The base case is 
trivial: we have T(0) < 2° = 1 (same for T(1)). For the inductive hypothesis: we assume that 
T(k) < 2*. We now use the inductive hypothesis to prove the statement for k + 1. Note that 


Tk +1) =T(k) + Tk-1)4+1< 2% 42%141 < 2% 42% = 2025 = 2441, 


where the last inequality follows from the fact that 2’! +1 < 2* for all k > 0. This com- 
pletes the inductive step and hence completes the proof. In conclusion, we have T(n) < 2” 
for n =0,1,2,.... E 


5.2 Recursion-lteration 


Within this section, we introduce the iteration method, a valuable technique for resolving 
recurrence relations. This method is characterized by a systematic approach where we dis- 
sect the given recurrence relation into a sequence of individual terms, each of which helps 
derive the expression for the nth term based on the preceding terms. This process continues 
iteratively until we reach the desired solution for the nth term of the recurrence. 

The iteration method is a powerful tool for solving recurrence relations, as it allows for 
a step-by-step and cumulative approach to understanding and deriving the expressions for 
the terms in the sequence. Through the examples presented in this section, we aim to pro- 
vide practical illustrations of how the iteration method is effectively applied, allowing read- 
ers to grasp its significance and applicability in solving various types of recurrence relations. 


Example 5.3 Use the iteration method to solve the following recurrence formula. 


TQ) = c, 


T(n) =c,+T(n-1), n>1, (5.3) 
where c, and c, are constants. 
Solution 
From (5.3), we have 
T(n) = co + T(n-1) 
= C + (c, + T(n — 2)) 
= Cy + (Cp + (C, + T(n — 3))) 
= 3c,+ T(n- 3) 
= ke, + T(n- k). 
Letk =n — 1, then n — k = 1 and we have 
T(n) = (n — 1)c, + TQ) = (n — 1)c, + c}. = 
Example 5.4 Use the iteration method to solve the following recurrence formula. 
T=: (5.4) 


T(n) = c+ T(n/2), 


where c is a constant. 
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Solution 
From (5.4), we have 


T(n) = c+T(n/2) 
= c+(c+T(n/4)) 
= c+ (c+ (c+ T(n/8))) 
= 4c + T(n/2") 


ll 


ck + T(n/2*), 
Let n=2*, then k=logn. It follows that T(n) =ck+T(n/2") =clogn+c, where 
c = T(n/2*) = T0). 7 
Example 5.5 Use the iteration method to solve the following recurrence formula. 

T(n) = n + 2T(n/2). (5.5) 


Solution 
From (5.5), we have 


T(n) = n+2T(n/2) 
= n+2(n/2+2T (n/4)) 
= n+2(n/2+2(n/4 + 2T (n/8))) 
= 3n + 2T (n/2°) 
= kn + 2*T (n/2*). 


Let n= 2", then k= logn. It follows that T(n) = nk + 2"T(n/2") = nlogn+nT(1) as 
desired. fe 


5.2.1 Change of Variables 

Sometimes we need to transform the recurrence to one that we solved before. We have the 
following example. 

Example 5.6 Solve the recurrence T(n) = 2T( yn) + logn. 


Solution 
Set m = logn, then n = 2” and 


T(2™) = 2T(2") + m. (5.6) 


After renaming m to k and letting S(k)= T(2"), the formula (5.5) becomes 
S(k) = 2S(k/2) +k, which is the same recurrence formula (5.6). In Example 5.5, we 
used the iteration method to solve this recurrence and found that S(k) = klogk + kS(1). 
It immediately follows that 


T(n) = T(?2”) = S(m) = mlogm + mS(1) = logn log(logn) + clog n, 


where c = T(2). | 
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5.3 Generating Functions 


Recurrence relations can be solved by finding a closed form for the associated generating 
function. 


Definition 5.2 Let {a,,} be a sequence of real numbers. The generating function for {a,} 
is the infinite series 


foe} 
g(x) = ao FOX + A,X? +++ H aX” +>: = Lax. 
k=1 


Below are some examples of generating functions. 


Example 5.7 The generating function for the sequence 1,1,1,... is 


co 
l 
Ll4+x4¢x?4---= N a = —— for |x| <1, 
gag h 
k=1 
where the last equality was obtained by the geometric series (3.5). a 


Example 5.8 The generating function for the sequence 1, a, a?, ... is 
foe} foe} 1 
Lt+axtart---= yaw = Yao = —— for |ax| <1, (5.7) 
k=1 k=1 ae 


equivalently, |x| < 1/|a| for a 4 0, where the last equality was obtained by the geometric 
series (3.5). E 


We can define generating functions for finite sequences of real numbers by extending 
a finite sequence a}, z, ...,Q„ into an infinite sequence by setting a; = 0 for j> n+ 1. 
We have the following example. 


Example 5.9 The generating function for the sequence ( ) : (") i (3) PNS (") is the 
finite series 


(Ae (Sytner 


k=1 


where the last equality was obtained by the binomial theorem (see Theorem 6.1). a 


In the generating function method for solving recurrences, we transform the recurrence 
relation for the terms of a sequence into an equation involving a generating function. 
This equation can then be solved to find a closed form for the generating function. The 
recurrences that we solve in the following example are taken from Rosen (2002) with little 
modification in their form. 


5.3 Generating Functions 


Example 5.10 Use the generating function method to solve the following recurrences. 


1, ifn =0 
8T(n — 1) +1071, ifn > 1. 


ifn=0 


n= a 4T(n-1), ifn> 1. 


(b) T(n) = { 


Solution 
(a) Let g(x) be the generating function for the sequence {a,}={T(n)}. Then 
g(x) = ar T(k)x*. Using the recurrence relation, we have 


g(x) = T(0) + rior! =4+ Sara -1x =4+ aS Tk —1)x*! 


k=1 k=1 


=4+ axy T(k)x*. 


k=0 
It follows that 
g(x) = 4 + 4xg(x). 


Solving for g(x), we get 


g(x) = Éro- is at ak 


e 


where the second equality was obtained by the geometric series (5.7). Therefore, 
T(n) = qn 


for any n > 0. 
(b) Let g(x) be the generating function for the sequence {a,,} = {T(n)}. Then 


g(x) = ET. 


k=0 


Using the recurrence relation, we have 


g(x) = TO) + X Tix" 


k=1 


1+ È GT- 1) +10 )x* 
k=1 


1+ p? T(k — 1)x®! + p2 10*-1yk-1 


ll 


Il 


> 


1+ aS Tb tid = = 1 + 8xg(x) t 


= ae 


where the last equality was obtained by the geometric series (5.7). It follows that 


g(x) = 1 + 8xg(x) +7 


=r. 
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Solving for g(x), we get 
1 x 1 — 9x 
80) = Gp ( tiz T a100 — 8x) 
Note that 


1 1 1 1 
1-9x==-(+1 8x + 10x) = =(1 — 8x) + =(1 — 10x). 
zí ) 3\ ) 56 ) ri ) 
Hence, the function g(x) can be written as 
5(1 — 8x) + 3(1 — 10x) 
(1 — 10x)(1 — 8x) 


_ 2, 1⁄2 1 ( i ¿1 ) 
1-10x 1-8 2\1-10x 1-8x/" 
Using the geometric series (5.7), we have 


g(x) 


= 1 ` k = ki < k k) y 
80) = 5 (Yow + 2,80 ) = 2i (10% + 8*) 
= = 
T(k) 
Therefore, 
T(n) = za +85) 
for any n > 0. E 


In summary, the generating function method is a powerful technique for resolving recur- 
rence relations. By breaking down these recurrences into a sequence ofindividual terms and 
deriving expressions for each term based on preceding ones, we can systematically unveil 
the solutions to these recursive problems. This approach not only helps us understand the 
underlying patterns but also provides a practical way to derive closed-form expressions for 
recurrence relations. 


5.4 Recursion-Tree 


The recursion-tree method is a strategy employed for tackling recurrence relations, and it 
involves a systematic approach. To utilize this method, we transform the given recurrence 
into a tree structure, where each node within the tree signifies the cost incurred at different 
levels of recursion. This tree effectively helps us visualize and break down the recurrence 
relation, allowing us to analyze and sum up the costs associated with each level of the 
recursion. 

In order to grasp the practical application of the recursion-tree method, it is often best 
comprehended through an illustrative example. The following example serves as a practical 
demonstration of how the method operates, offering a clear and tangible instance of how 
the tree structure is employed to calculate and sum the costs at various recursion levels. 
Through this illustration, readers can gain a deeper understanding of the methodology 
and its utility in resolving recurrence relations. Example 5.11 is extracted with some 
modifications from (Cormen et al., 2001, Section 4.4). 


5.4 Recursion-Tree 


Example 5.11 Use the recursion-tree method to determine a good upper bound on each 
of the following recurrences. (Here c is a constant.) 


(a) 
(b) 
(c) 


T(n) = 2T(n/2) + n?. 
T(n) = 3T(n/4) + cn’. 
T(n) = T(n/3) + T(2n/3) +n. 


Solution 


(a) 


(b) 


For convenience, we assume that n is an exact power of 2 so that all subproblem sizes 
are integers. In Figure 5.1, we show in detail the construction of a recursion-tree for the 
recurrence T(n) = 2T(n/2) + n?. Note that we continue expanding each node in the tree 
by breaking it into its constituent parts as determined by the recurrence. 

Let h be the height of the recursion-tree. At each level i = 0,1,2,...,h, we have 

e subproblem size = n/2', 

e cost of each node = (n /25?, 

e number of nodes = 2’. 

The total cost at all levels is T(n) = ae (total cost at level i), where 


total cost at level i = (# of node at level i) -(# cost of each node at level i) 


2 2 
_sifn\ an 
-2(4) =e 


Now, we want to know how to find the height of the tree h. Because subproblem sizes 
decrease by a factor of 2 each time we go down one level, we eventually must reach a 
boundary condition. Essentially, we want to know how far from the root do we reach 
one. Note that the subproblem size for a node at level h is n/2". Therefore, starting from 
the root, we reach one when n/ 2" = 1 (as T(n / 2") = T(1)). Hence, the height of the tree 
ish = logn. 
As a result, the total cost at all levels is 
beng pes 1 i r oœ 1 i si A 
mer mi (5) =a (>) sn 421 = 


2 


where the last inequality follows from the geometric series (3.5). Thus, T(n) < 2n?. 
For convenience, we assume that n is an exact power of 4 so that all subproblem 
sizes are integers. In Figure 5.2, we show the recursion-tree for the recurrence 
T(n) = 3T(n/4) + cn?. 

Let h be the height of the recursion-tree. At each level i = 0,1,2,...,h, we have 

e subproblem size = n/4, e number of nodes = 3/. 

e cost of each node = c(n/4')?, 


The total cost at all levels is T(n) = x (total cost at level i), where 
total cost at level i = (# of node at level i) - (cost of each node at level i) 
2 i 
(4) 3! = on?(=) f 
4! 16 


To know how far from the root do we reach one, we note that the subproblem size for a 
node at level h is n/4". Therefore, starting from the root, we reach one when n/4" = 1. 
Hence, the height of the tree is h = log,n. 
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T(n) = 2T(n/2) + n? 


T(n/2) = 4T(n/4) + 2(n/2)? 


T(s) r(e) Th) r(e) Th) r(5)7(8) (5) 


TO = TO TQ) = TQ) TA) - TQ) TQ) - TQ) 


Figure 5.1 Constructing a recursion-tree for T(n) = 2T(n/2) + n°. 


5.4 Recursion-Tree 


e(a) (BY (BY (BY ov eB (BY ow 


TQ) ++ TATA) + OT = TAYTG) + TAY) + TAYTA) ++ TA) 


Figure 5.2 The recursion-tree for T(n) = 37(n/4) + cr’. 


As a result, the total cost at all levels is 


log,n è co ; 
2/3 V 2 3 V 2 1 16 2 
T(n) = 2e (=) <cn >>) cn (2) T 
16 
where the last inequality follows from the geometric series. Thus, we have T(n) < 
(16/3)cn?. 
(c) Figure 5.3 shows the recursion-tree for T(n) = T(n/3) + T(2n/3) + n. Note that, unlike 
in the trees of (a) and (b), in this tree the paths from the root to the leaves have dif- 
ferent lengths. Let h be the height of the recursion-tree. Note also that the total cost 


I 
= 


x=-----------5; 


‘on S4 —n 
N a 
: ` 
j $ 
n SS 2n zA 
d oA = $ —n 
3 Sa 3 aS 
ee S Da 
eae OS, 3 
A ` 
x 3 
n 2n 2n Sy 4n $ 
as owes paie Sa 5 — Nn 
9 9 9 ` 9 Xa 


TA) = TA TA = TA) TA) = TA) TA) = TA 


Figure 5.3 The recursion-tree for T(n) = T(n/3) + T(2n/3) +n. 
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at each level i(i = 0,1,2,...,h) equals n, and that the longest path from the root to a 
leaf is 


n— (2)n — (2) n— (2) n—--- 1. 


Starting from the root, we reach one when (2/3)'n = 1. Hence, the height of the tree is 
h = log, zn. It follows that the total cost at all levels is 


log, 20 

logn 

T(n) < n = (1+ log; pn) n= (1+ ) n < kn logn, 
2 log 3/2 


for some positive constant k. Thus, T(n) < knlogn. a 


Note that in Example 5.11 (c), we were looking for the longest path from the root to a leaf 
because we need an asymptotic upper bound. However, if an asymptotic lower bound is 
required, we should be looking the shortest path from the root to a leaf as in Exercise 5.9 (b). 


Exercises 


5.1 Choose the correct answer for each of the following multiple-choice questions/ 
items. 
(a) If the guess-and-confirm method is used to solve the recurrence relation 
T(n) = yn T( yn) +n, we find that T(n) is (here c is some constant) 


(i) en. (ii) cnv/log n. (iii) en logn. (iv) en log(log n). 


(b) If we use the iteration method to solve the recurrence. 


T(n-1)+n, ifn>1; 
r= {ee 


> 


we find that T(n) is 
(i) 14+2---+(n-2)+(n—-1) =n(n-1)/2. 
(ii) 1+24+---+(n-1)+n=n(n+4+1)/2. 
(iii) 1+24+---+n+(n4+1)=(n+1)(n4+2)/2. 
(iv) 1+2+---+n+n? =n(3n+1)/2. 
(c) The generating function for the sequence 1, —1, 1, —1, 1, —1, ... is 


s 1 oss 1 
G) Dox = — for |x| <1. Gn) > (= Ty for [xl <1. 


xX 


i ©% k_ 1 . kt+1yk — 1 
Gi) Eror = zz for xl <1. (iv) Vey (1x = Ta for |x| <1. 


(d) The height of the recursion-tree drawn to give a good asymptotic upper bound 
on the recurrence relation T(n) = T(n/5) + T(4n/5) + nis 


(i) log,n. (ii) log;n. (iii) log. /4n. (iv) log,,5n. 


5.2 


5.3 


5.4 


5.5 


5.6 


5.7 


5.8 


5.9 
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(e) The height of the recursion-tree drawn to give a good lower bound on the recur- 
rence relation T(n) = T(n/3) + T(2n/3) + nis 


(i) log, /3N. (ii) log,n. (iii) log, .n. (iv) log,n. 


Use the guess-and-confirm method to solve the following recurrences. 
(a) T(n) = 2T(n/2) + n, and T(1) = 1. (b) T(n) = 3T(n — 1) + 4, and T(1) = 1. 


Use the guess-and-confirm method to solve the following recurrence. 
1, ifn = 0; 
T(n)=4 0, if n = 1,2; 
5T(n — 1) — 8T(n — 2)+4T(n —- 3), ifn >23. 


Use the iteration method to solve the following recurrences. 
(a) T(n) = 5T(n — 1), where T(0) = 3. 
(b) T(n) = 2T(n/2) + nlogn, where T(1) = c for some constant c. 


Use the generating function method to solve the following recurrence. 
1, if n = 0; 
T(n)=< 3, ifn = 1; 
-T(n-1)+6T(n-2), ifn>2. 


Use the generating function method to solve the Fibonacci recurrence: 
0, ifn =0; 
T(n) =4 1, ifn = 1; 
T(n—1)+T(n-2), ifn>2. 


Use the recursion-tree method to solve the recurrence T(n) = 2T(n — 1) + 1. 


Use the recursion-tree method to determine a good upper bound on each of the 
following recurrences. 


(a) T(n) = T(n/2) +n’. (b) T(n) = T(n — 1) + T(n/2)+ n. 


Use the recursion-tree method to determine a good lower bound on each of the 
following recurrences. 

(a) T(n) = 4T((n/2) + 2) +n. 

(b) T(n) = T(n/3) + T(2n/3) + cn, where c is a constant. 
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5 Recurrences 
Notes and Sources 


The study of recurrences, or recurrence relations, has a long history dating back to ancient 
civilizations. One of the earliest known instances of recurrence relations is in the work of 
Fibonacci, an Italian mathematician from the 13th century, who introduced the famous 
Fibonacci sequence to model the population growth of rabbits; see Sigler (2003). Another 
noteworthy historical figure in the development of recurrence relations is Blaise Pascal, 
who investigated Pascal’s triangle, a triangular array of numbers that exhibits numerous 
recurrence patterns, in the 17th century; see Pascal (1665). 

In this chapter, we learned several recurrence-solving techniques, like guess-and- 
confirm, recursion-iteration, generating functions, and recursion-tree. As we conclude 
this chapter, it is worth noting that the cited references and others, such as Rosen (2002); 
Cusack and Santos (2021); Mott et al. (1986); Pottenger (1997); Spencer (1994); Lueker 
(1980); Bentley et al. (1980); Grimaldi (1999); Liu (1968), and Greene and Knuth (2009), 
also serve as valuable sources of information pertaining to the subject matter covered in 
this chapter. Exercise 5.9 is due to Cormen et al. (2001). 
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Readers after this chapter will gain the necessary tools and knowledge required for profi- 
cient counting and enumeration. The first section of this chapter introduces the binomial 
series, which is needed before we start with the counting methods. This also includes the 
essence of a combinatorial proof and some binomial coefficients and identities. 


6.1 Binomial Coefficients and Identities 


A polynomial represents an algebraic expression comprising variables and coefficients, 
where the permitted operations include addition, subtraction, multiplication, and the 
nonnegative integer exponentiation of variables. For example, x? + 4xyz? + 3y’z is a 
polynomial of three variables. A polynomial of one variable has the form 


P(x) = a,x" + apx"! + +++ + a,x? + ax + dp, 


where n is a nonnegative integer and the constant numbers ap, a,,a,,...,a,, are the coef- 
ficients. A monomial is a polynomial which has only one term. For example, 2x, 4x? and 
5x’y* are monomials. A binomial is a polynomial that is the sum of two terms, each of 
which is a monomial. For example, 2x + y is a binomial. The function 3x? + 4y3z is also a 
binomial where the constants 3 and 4 are the binomial coefficients. A term can be a product 
of constants and variables, such as 5x”y*z?, but that does not concern us in our context here. 
In this section, we establish some identities that express relationships among the so-called 
binomial coefficients. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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6 Counting 


Most proofs that are given in this section are combinatorial proofs. A combinatorial proof 
of an identity is a proof that uses either two types of mathematical proof: 


e A double counting proof: A proof that employs counting principles to demonstrate that 
both sides of the identity enumerate the same objects, albeit through distinct methods. 

e A bijective proof: A proof that relies on establishing the existence of a one-to-one corre- 
spondence (bijection) between the sets of objects counted by the two sides of the identity. 


An example of a combinatorial proof is the proof of Theorem 6.1 presented in 
Subsection 6.1.1, which is variously known as the binomial theorem. 


6.1.1 The Binomial Theorem and Coefficients 


The binomial theorem gives a formula for expanding the binomial power (x + y)” for any 
positive integer n. The symbol (z) that appears in the following theorem is read as “n 


choose k,” where n and k are integers that satisfy n > k > 0. The symbol (z) represents 
the number of ways to choose k objects from a set of n objects and is given by the formula! 
(") = n! -nn-D--(n-k+1) 
k k\n—k)! k(k-1)---1 


It is easy to see that, for any integers n > k > 0, we have 


ie) a a a 


Note also that C) =k and (K) = = 1 for any integer k > 0. This includes (G) = 


0 
A 

The symbol : ) occurs as coefficients in the binomial theorem stated below. These coef- 
ficients are commonly called the binomial coefficients. 


Theorem 6.1 (The binomial theorem) Let x and y be variables and n be a nonnega- 
tive integer, then 


a+ =X (z) = (oer (Txt te (joo te (2 ly. 


Proof: When the product is expanded, its terms take the form of x”-*y* for values of k rang- 
ing from 0 to n. To determine the count of terms in this form, consider that in order to 
obtain such a term, you must select n — k instances of x from the total of n terms (leaving 
the remaining k as y terms). Hence, the coefficient of xrkyk is ( 7i y ), which is equivalent 


(r): E 


The series presented in the binomial theorem is known as the binomial series. Some com- 
putational uses of the binomial theorem are illustrated in the following examples. 


1 The factorial of a positive integer n, denoted by n!, is defined as n! = n(n — 1) - - - 2-1. For example, 
4! = 4-3-2-1 = 24. Note that 0!=1 and 1!=1. 


6.1 Binomial Coefficients and Identities 


Example 6.1 Find the expansion of each of the following expressions. 
(a) @+y)’. (b) @+y)*. (c) œ +y). 


Solution 
(a) From the binomial theorem, it follows that 


(ty)? = x (7)! 


k=0 
= (5) + (7) y+ G07 + (5) = 2 + ey + 307 +". 


The same result can be obtained by simple computations as follows. 
yY = x+y? = (K+? + Wy y) =x? + 3x*y + Bry’ +y?. 


(b) From the binomial theorem it follows that 


aay = > (xo 


(e 


= x4 + 4y + 6x?y? + AY? +y. 


ll 


(c) The solution of this item is left as an exercise for the reader (see Exercise 6.2). E 


Example 6.2 Find the coefficient of x’y”” in each of the following expressions. 
(a) &+y)”. (b) (3x — 2y)”. 


Solution 
(a) From the binomial theorem, it follows that 
! 
FA = MA = 50, 388. 
12 12! 7! 
(b) By the binomial theorem, we have 
19 


(3x — 2y)? = (3x + (-2y))? = J, ( 


k=0 


19 


p Goz. 


Consequently, the coefficient ofx’y!? in the expression is obtained when k = 12, namely 


! 
‘eg 37(-2)% = 1! 372 451,373, 285,376. 
12 12! 7! m 


6.1.2 Binomial Identities 


The binomial theorem serves as a powerful tool in mathematics, offering a method to 
establish and validate numerous binomial identities. By leveraging this theorem, mathe- 
maticians can unravel and confirm relationships among binomial expressions, providing 
a systematic approach to understanding the structure and properties of these identities. 
The following corollary, derived from the binomial theorem, contributes to the depth and 
applicability of binomial identities in mathematical exploration and problem-solving. 


151 


i) 6 Counting 


Corollary 6.1 Let n be a nonnegative integer. We have 


a oe iyn E £ n 
P © F(a) = Z (aa): 


s "O ra 
(b) Le) =0. (d) Y(t) = 3", 


Proof: 
(a) Using the binomial theorem with x = 1 and y = 1, we have 


n 


|: SPN n_ n n-K _ n 
2"=(141) x) k A 


k=0 
(b) Using the binomial theorem with x = —1 and y = 1, we have 


0=0" =((-1)+1)"= 5 (Z) cpa = 5 (Z) cot. 
k=0 k=0 
(c) From item (b), we have 
WD ER R a 
and hence 
e as G ete 


This proves item (c). 
(d) The proof of this item is left as an exercise for the reader (see Exercise 6.4). E 


The binomial coefficients satisfy some important recurrences and identities. We first 
introduce the so-called Pascal’s identity or Pascal’s recurrence formula, which is defined 
as T(n,k) = T(n — 1,k — 1) + T(n — 1, k), where T(n, 0) = T(n,n) = 1. It can be shown that 
the solution of this recurrence is the binomial coefficient T(n, k) = (a We have the 
following theorem. 


Theorem 6.2 (Pascal’s identity) Letn and k be integers with n > k > 0. Then 
n\_/n-l1 n-1 
Aaa y): 
Proof: Note that 
n-1 n-1 (n-1)! (n—1)! 
aes, ( k ) @-Oik-1 mole 
___@™-D!__k, (=! n-k 
~ (n=-k)!(k-1)!k (n-1-bk!k!n-k 
_ (n-1!k (n-1!(n-k) 
(n—k)!k! (n—k)!k! 
7 oane aam 
~ (k+n-k)!k!  \k/’ 
This proves the theorem. E 


6.1 Binomial Coefficients and Identities 


Figure 6.1 Pascal's triangle. 


Pascal’s identity is the basis for a geometric arrangement of the binomial coefficients in 
a triangle, as shown in Figure 6.1. 
The ith row in the triangle consists of the binomial coefficients 


(0): Ga) (a) = (i): 


This triangle is known as Pascal’s triangle which is due to Blaise Pascal (1623-1662). Pascal’s 
identity illustrates that when you add two consecutive binomial coefficients within this 
triangular array, the result is the binomial coefficient found in the next row, positioned 
between these two coefficients. 

The identity in the following theorem is due to Alexandre-Theophile Vandermonde 
(1735-1796). 


Theorem 6.3 (Vandermonde’sidentity) Let m, n, and r be nonnegative integers with 
r not exceeding either m or n. Then 


Cer) 


k=0 


Proof: Using the binomial theorem, we have 


m 


Sae =(+x"" 


r= 


= (1+x)"(1+x)" 


(EC) 4) BEM) 


where we used (3.6) to obtain the last equality.” 


2 Note that (3.6) can be used because the binomial coefficients ( A ) and ( ) give zero for all i > m and 
j > n, respectively. 
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By comparing coefficients of x”, the identity follows for all integers r with 0 < r < m + n. 
For larger integers r, both sides of the identity are zero due to the definition of binomial 
coefficients. The proof is complete. a 


The following is a corollary of Vandermonde’s identity. 


Corollary 6.2 For any nonnegative integer n, we have 
(7) =D) 
n A \k 


Proof: The result follows immediately from Vandermonde’s identity with m = r = n and 
the identity ( i ) = ( oy ). m 


6.2 Fundamental Principles of Counting 


Counting has applications in a variety of areas, including computer science, probability, 
and statistics. In this section, we present some fundamental counting principles. 


6.2.1 The Product Principle of Counting 


We start with the product principle which is stated below. 


Principle 6.1 Assume there is a procedure that can be broken down into a sequence of 
two tasks. If there are n, ways to do the first task and for each of these ways of doing the first 
task, there are n, ways to do the second task, then the procedure can be done inn, X n, ways. 


The following example shows how the product principle is used. 


Example 6.3 In the country of Jordan, there are some amazing restaurants worth to visit 
in some cities, such as Amman city, Petra city, and Jerash city. Suppose that one can take 
three routes from Jerash to Amman, and can take five routes from Amman to Petra. See 
Figure 6.2. How many possible routes can one take to get from Jerash to Petra if every 
Jerash—Petra route consists of two routes that are linked in Amman? 


Solution 

Every Jerash—Petra route consists of a Jerash-Amman route, which can be done using three 
possible routes, and an Amman-Petra route, which can be done using five possible routes. 
Therefore, by the product principle, there are 3 x 5 = 15 possible routes can one take to get 
from Jerash city to Petra city. a 


The following theorem is the extended version of Principle 6.1. 


Theorem 6.4 Assume that a procedure can be broken down into a sequence of m tasks, 
say T,,T>,...,T,,. If each task T,,i = 1,2,...,m, can be done in n; ways, regardless of how 
the previous task was done, then the procedure can be done inn, X n, X + + X n,, Ways. 


6.2 Fundamental Principles of Counting 


° 
Ruwayshid 


e 
Jerash 


® Amman 


Amman 


Jordan 


Saudi Arabia 


Figure 6.2 The possible routes that one can take to get from Jerash city to Amman city, and those 
to get from Amman city to Petra city in the country of Jordan. 


Theorem 6.4 can be proved using induction on m (see Exercise 6.7). The following examples 
show how Theorem 6.4 is applied. 


Example 6.4 Each Jordanian national ID card contains an ID number in the back, which 
consists of three uppercase English letters followed by five digits. What is the number of 
possible Jordanian national ID cards? 


Solution 

There are 26 choices for each of the 3 uppercase English letters and 10 choices for each of 
the 5 digits (see the graph shown below). Hence, by Theorem 6.4, the number of possible 
Jordanian national ID cards is 


26 x 26 x 26x 10x 10x 10x 10x 10 = 1,757, 600, 000. 


26 choices for each letter. 10 choices for each digit. 
a 


Example 6.5 Use the product principle of counting to prove that the cardinality of the 
powerset of a finite set A is equal to 2” if the cardinality of A is n.’ 


Solution 

When constructing a subset of the set A, it involves performing n distinct tasks, where each 
task entails making a decision about whether to include a particular element in the subset. 
As each task has two possible outcomes (either including or excluding the element), there 


3 The reader is asked to prove this fact in Exercise 2.4 by mathematical induction. 
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are a total of 2” ways to complete this process. Consequently, the set of all possible subsets 
of A, that is P(A), contains precisely 2” elements. This concludes the intended proof. m 


Example 6.6 What is the value of “sum” after the fragment* given in Algorithm 6.1, 
where n, m, p, and q are positive integers, has been executed? 


Solution 
Let T, be the number of times of traversing the statement (and only the statement) given in 
line (s) for each s = 1, 2,...,6. 


Algorithm 6.1: The algorithm of Example 6.6 
1: sum = 0 

2: for (i= 1;i < n;i + +) do 

3: | for (j = 0;j < m;j++)do 

4: for (k = p;k > 1;k-——) do 

5: for (r = q;r > 0;r — —) do 

6 | sum = sum +1 


7: end 
8: end 
9: | end 

10: end 


The number of times of traversing the statement given in line (s) is the number of ways 
to do the task T, for s = 1,2, ...,6. These numbers are listed in Table 6.1. 

Also, let P, be the procedure of executing the body of the “for” loop in line (r) for 
each r = 2,...,5. Let us add a column for the number of ways to carry out the procedure 
P,,,U = 2,...,5 (see the side comments in Algorithm 6.2). The initial value of “sum” is zero. 
Each time the “sum” statement in line (6) is executed, 1 is added to “sum.” Therefore, the 
value of “sum” after the fragment given in Algorithm 6.1 has been executed is equal to the 


Table 6.1 Numbers of ways to do the tasks of going over simple and for statements. 


Line Statement Task # ways to do task T, 
1 sum=0 Ti 1 
2 for (i=1; i<=n; i++) T, n 
3 for (j=0; j<m; j++) T; m 
4 for (k=p; k>=1; k—-) T, p 
5 for (r=q; r>0; r—-) T; q 
6 sum=sum+1 Ts 1 


4 “For-statement” written in the fragment is formally introduced in Section 7.1.1. 
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number of times to execute the body of the “for” loop in line (5), which is the number of 
ways to carry out the procedure P.. 


Algorithm 6.2: Algorithm 6.1 revisited 


1: sum = 0 
2: for (i = 1;i < n;i + +) do // Procedure P, is done in n ways 
3: | for (j = 0;j < m;j++)do // Procedure P, is done in n x m ways 
4: for (k = p;k > 1;k — —) do // P, is done inn x m x p ways 
5 for (r = q;r > 0;r — —) do // P; is done in n x m x p x q ways 
6 | sum = sum + 1 
7: end 

end 
9 | end 
10: end 


The following points are noted, which lead us to conclude the number of ways to carry 
out the procedure P.. 


e How many times do we execute the body of the “for” statement in line (2)? The answer is 
from 1 to n, which is n times. Note that when i = n + 1 is checked, the inequality is false. 
Hence, the number of ways to carry out the procedure P, is n. 

e How many times do we execute the body of the “for” statement in line (3)? If the “for” 
statement in line (3) was not nested, it would execute m times just like the “for” statement 
in line (2). Since the “for” statement in line (3) is nested, its body is executed m times for 
each time we execute the body of the “for” statement in line (2) (which is n times). In 
other words, by the product principle of counting, the number of ways to carry out the 
procedure P, is equal to the number of ways to do the task T, multiplied by the number 
of ways to do the task T}, which is n x m. 

e Similarly, since the “for” statement in line (4) is nested, its body is executed p times for 
each time we execute the body of the “for” statement in line (3) (which is n x m times). 
In other words, by the product principle of counting, the number of ways to carry out the 
procedure P, equals the number of ways to do the task T, times the number of ways to 
do the task T, times the number of ways to do the task T} which is n x m x p. 

e Again and similarly, since the “for” statement in line (5) is nested, its body is executed q 
times for each time we execute the body of the “for” statement in line (4) (which is n x 
m x p times). In other words, by the product principle of counting, the number of ways to 
carry out the procedure P, equals the number of ways to do the task T, times the number 
of ways to do the task T, times the number of ways to do the task T, times the number of 
ways to do the task T;, which is n x m x p X q. 


Thus, the value of “sum” after the fragment given in Algorithm 6.1 has been executed is 
equal ton x m xp xq. E 


The product principle is also given in terms of sets. We have the following remark. 
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Remark 6.1 The number of elements in the Cartesian product of finite sets is the product 
of the number of elements in each set. 


Proof: Let A,,A,,...,A,, be finite sets. Note that the task of choosing an element in the 
Cartesian product A, X A, X---XA,, is done by choosing an element in A,, an element in 
A,,...,and an element in A,,. By the product principle, it follows that 


|A XA, X: XA |A,| X [A| X X Aml- 


m| = 


The proof is complete. a 


6.2.2 The Sum Principle of Counting 


In this part, we present the sum principle of counting. 


Principle 6.2 Ifa task can be done either in one of n, ways or in one of n, ways, where 
none of the set of n} ways is the same as any of the set of n, ways, then there are n, + n, 
ways to do the task. 


The following example shows how the sum principle is used. 


Example 6.7 Sara has decided to study a Bachelor’s degree in Mathematics at one of the 
Jordanian universities, either in Irbid city or Amman city. If Sara decides to go to Irbid, she 
will study at either Yarmouk University, Jordan University of Science and Technology, Irbid 
National University, or Jadara University. If Sara decides to go to Amman, she will study at 
either University of Jordan, Philadelphia University, Petra University, Al-Zaytoonah Uni- 
versity, or Al-Ahliyya Amman University. What is the possible number of Jordanian uni- 
versities Sara could choose? 


Solution 

There are four possible universities (four ways) Sara could go to Irbid city, and five possi- 
ble universities (five ways) Sara could go to in Amman city. Thus, by the sum principle of 
counting, there are 4+ 5 = 9 possible universities Sara could choose to study in Irbid or 
Amman. E 


The following theorem is the extended version of Principle 6.2. 


Theorem 6.5 Assume that a procedure can be done in one of n, ways, in one of n, ways, 
...,orin one of n,, ways, where none of the set of n; ways of doing the procedure is the same 
as any of the set of n, ways, for all pairs i and j with 1 < i <j < m, then the total number of 
ways to do the procedure is n} + n, +: + + + Nnm ways. 


Theorem 6.5 can be proved using induction on m (see Exercise 6.8). The following 
examples show how Theorem 6.5 is applied. 


Example 6.8 Zaid won a scholarship to study an undergraduate degree at one of the 
following universities in Ohio: Case Western Reserve University (CWRU), Kent State 
University (KSU), Ohio State University (OSU), and University of Cincinnati (UC). 
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At CWRU, the scholarship requires that Zaid chooses either veterinary medicine, biomed- 
ical engineering, or biology. At KSU, it requires that Zaid studies clinical nutrition. At 
OSU, it requires that Zaid chooses either medicine, dentistry, pharmacy, or public health. 
At UC, it requires that Zaid chooses either nursing or chemistry. How many possible 
undergraduate degree programs that Zaid can choose from? 


Solution 

Zaid can choose an undergraduate degree program by selecting a program at CWRU, KSU, 
OSU, or UC. There are three ways to choose a program at CWRU, one way to choose a 
program at KSU, four ways to choose a program at OSU, and two ways to choose a program 
at UC. Therefore, by the sum principle of counting, there are 3 + 1 + 4+ 2 = 10 ways to 
choose a program. a 


Example 6.9 Whatis the value of “sum” after the fragment given in Algorithm 6.3, where 
n,m, p and q are positive integers, has been executed? 


Solution 

Let T, be the task of executing the statement in line (s) for each s=1,2,3,5,6,8,9, 
11,12. Let us add comments showing the number of ways to carry out the task T,, 
s=1,2,3,5,6,8,9, 11,12 (see the comments in gray in Algorithm 6.3). The initial value 
of “sum” is zero. For each time each “sum” statement in lines (3), (6), (9), and (12) is 
executed, 1 is added to “sum.” Therefore, by the sum principle of counting, the value of 
“sum” after the fragment given in Algorithm 6.3 has been executed is equal to the number 
of ways to carry out the task T,, plus the number of ways to carry out the task T,, plus the 
number of ways to carry out the task T}, plus the number of ways to carry out the task T;,. 


Algorithm 6.3: The algorithm of Example 6.9 


1: sum = 0; // Task T, is carried out in 1 way 
2: for (i = 1;i < n;i + +) do // Task T, is carried out in n + 1 ways 
3: | sum = sum + l; // Task T, is carried out in n ways 
4: end 
5: for (j = 0; j < m; j + +) do // Task T, is carried out in m + 1 ways 
6: | Sum=sum+1; // Task T, is carried out in m ways 
7: end 
s: for (k = p;k > 1;k — —) do // Task T, is carried out in p + 1 ways 
9 | sum = sum + 1; // Task T, is carried out in p ways 
10: end 
11: for (r = q;r > 0;r — —) do // Task T,, is carried out in q + 1 ways 
12: | sum = sum +1; // Task T,, is carried out in q ways 
13: end 


Note that the number of ways to carry out the task T, equals the number of times to 
execute the body of the “for” statement in line (2). How many times the “for” statement 
in line (2) executes? 1 to n, which is n times, plus 1 for the last time that i is checked and 
the inequality is false, that is, n + 1 times. Hence, the number of times the statement in 
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line (3) (i.e., body of the “for” statement in line (2)) is n. Thus, the number of ways to carry 
out the task T; is n. Similarly, the tasks T,, Tọ, and T,, are carried out in m, p, and q ways, 


respectively. 
Therefore, by the sum principle of counting, the value of “sum” after the fragment given 
in Algorithm 6.3 has been executed is equal ton + m +p +q. E 


The sum principle is also given in terms of sets. We have the following remark. 


Remark 6.2 The number of elements in the union of pairwise disjoint finite sets is the 
sum of the number of elements in each set. 


Proof: Consider the sets A,,A,,...,A,,, Which are pairwise disjoint, meaning that their 
intersections satisfy A; Nn Aj = Ø for all i,j. It is important to note that within each set A,, 
there are precisely |A;| ways to select an element, where i ranges from 1 to m. Due to the 
pairwise disjoint nature of these sets, selecting an element from one set, say A,, does not 
simultaneously entail selecting an element from any other set, such as A;. Consequently, in 
accordance with the sum principle, since we cannot select an element from two different 
sets simultaneously, the total number of ways to choose an element from the union of these 
sets, denoted as |A, UA, U + + U Aml» is given by the sum of the individual set sizes: 


|A,UA,U---UA |A l+ lAl ++ Anl: 


m| = 


This equality holds true when A; N A, = Ø for all i,j. The proof is complete. a 


6.2.5 The Subtraction Principle of Counting 


In this part, we present the subtraction principle which is also known as the principle of 
inclusion-exclusion. 


Principle 6.3 Ifa task can be done either in one of n; ways or in one of n, ways, then the 
total number of ways to do the task is n, + n, minus the number of ways to do the task that 
are common to the two different ways. 


The subtraction principle is also given in terms of sets. The following remark follows 
immediately from Principle 6.3. 


Remark 6.3 (Inclusion-exclusion for two sets) The number of elements in the 
union of finite sets A, and A, is the sum of the number of elements in each set, minus the 
number of ways to select an element that is in both A, and A,. That is, 


|A,; UA = [Al + [AQ] — |A, N Ad]. (6.2) 
We have the following example. 


Example 6.10 Let S be a space or universe of two sets A and B, where |S] = 60, |A| = 
34, |B| = 22 and |A n B| = 8. Find |(A UB)’. 


6.3 The Pigeonhole Principle 


Solution 
By using (6.2), we have 


(AU B)'| = |S| — |A U B| = [S| — (|A| + |B] — |A A BI) = 60 — (34 + 22 - 8) = 12. 


This responds to the example’s query. a 


Note that Equation (6.2) can follow directly from Venn diagrams. Also note that the fol- 
lowing basic formula is also followed directly from Venn diagrams. 


|A —A,| = lAl z |A: Nn Al. 


The subtraction principle can be extended to find the number of ways to do one of n 
different tasks or, equivalently, to find the number of elements in the union of n finite sets 
(see Rosen (2002) for more detailed presentations). 

We have introduced the product, sum, and subtraction principles of counting. There is 
also a division principle of counting which is out the scope of this book. See Rosen (2002) 
for a good presentation of the division principle of counting. 


6.3 The Pigeonhole Principle 


Consider a situation in which there are 10 pigeons and only 9 pigeonholes available. In such 
a scenario, it is a certainty that at least one of the 9 pigeonholes will contain a minimum 
of two pigeons. To grasp the reason behind this assertion, observe that if every pigeonhole 
could accommodate, at most, a single pigeon, then the total number of pigeons that could 
be accommodated would be limited to a maximum of 9 (with one pigeon per hole). See 
Figure 6.3. 

The pigeonhole principle can be stated as follows. 


Principle 6.4 Ifthere are more pigeons than pigeonholes, then there must be at least one 
pigeonhole with at least two pigeons in it. 


D & A S 
A | Ss @s| & 
S & 


D S 
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Figure 6.3 Illustrating the pigeonhole principle: There are 10 pigeons but only 9 pigeonholes, at 
least one of these 9 pigeonholes must have at least 2 pigeons in it. 
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More generally, we have the following theorem. 


Theorem 6.6 Let k be a positive integer. If k + 1 or more objects are placed into k boxes, 
then there is at least one box containing two or more of the objects. 


Proof: Suppose, on the contrary, that none of the k boxes contains more than one object. 
Then the total number of objects would be at most k. This contradicts the fact that there are 
at least k + 1 objects. E 


The following examples show how Theorem 6.6 is applied. 


Example 6.11 If20 different algorithms exist to solve 19 different problems, then there is 
at least one problem that can be solved by two different algorithms. E 


Example 6.12 Because there are only 366 possible days in the year, among any group of 
367 people, there must be at least two of them with the same birthday. a 


The following theorem is the extended version of Theorem 6.6. 


Theorem 6.7 Let n and k be positive integers. If k(n — 1) + 1 or more objects are placed 
into k boxes, then there is at least one box containing n or more objects. 


Proof: Suppose, in the contrary, that none of the k boxes contains more than n — 1 objects. 
Then the total number of objects would be at most k(n — 1). This contradicts the fact that 
there are a total of k(n — 1) + 1 objects. a 


Another version of Theorem 6.7 is the following theorem, which can also be proved using 
contradiction (see Exercise 6.9). 


Theorem 6.8 Let mand k be positive integers. If m objects are placed into k boxes, then 
there is at least one box containing at least [m/k] objects.° 


The following examples show how Theorems 6.8 and 6.7 are applied. 


Example 6.13 As per the definition provided by the Pew Research Center, a millennial 
is characterized as an individual born within the timeframe spanning from 1981 to 1996. 
Consequently, in the year 2021, this would encompass millennials ranging in age from 24 
to 40. Among a group of 145 millennials, it is guaranteed that there are at least 10 individ- 
uals who share the same birth year. This can be seen by applying Theorem 6.7, taking into 
account the equation 145 = 16(10 — 1) + 1. E 


5 For a real number x, the ceiling of x is denoted as [x] and is defined as the smallest integer that is not 
smaller than x. For example, [5] = 5 and [v5] = 3, 
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Example 6.14 A Nobel Prize-winning scientist was invited to give plenary talks at 25 
international conferences which were held in 2019. From Theorem 6.8, among these 25 
conferences, there are at least [25/12] = 3 which were held in the same month. a 


The pigeonhole principle is also given in terms of sets. We have the following remark. 


Remark 6.4 Let k bea positive integer. If A is a finite set with k + 1 or more elements and 
Aj, A), ...,A, are subsets of A that form a partition of A, then there exists at least one subset 
A,, where i € {1,2,...,k}, such that |A,| > 2. 


Rather than proving Remark 6.4, we state and prove the following theorem which gener- 
alizes Remark 6.4. 


Theorem 6.9 Letr,,r,,...,1, be positive integers. If A is a finite set with Sh ry-k+1 
or more elements and Aj, A,, ...,A, are subsets of A that form a partition of A, then we have 
|A;| > r; for some i € {1,2,...,k}. 


Proof: We prove the theorem by contradiction. Suppose that for each i € {1,2,...,k} we 
have that |A,| < r; — 1. By the sum principle, we have 


|A] = [A4] + lAl +: + lAl 
< (r1 1)+(r-1)+: +0- 1) 


(3) 


which contradicts the fact that A has more than OG ri) — k elements. Thus, |A;| > 2 for 
some i € {1,2,...,k}. The proof is complete. E 


Note that Remark 6.4 is a special case of Theorem 6.9 which occurs when r, = r, =--- = 
ry, = 2. 

Ramsey Theory refers to the study of partitions of large structures, and generalizes the 
pigeonhole principle. The Ramsey number R(m, n) gives the solution to the party problem, 
which asks for the minimum number of guests R(m, n) that must be invited so that at least m 
will know each other or at least n will not know each other. For example, one can show that 
RG, 3) = 6. By symmetry, it is true that R(m, n) = R(n, m). It is possible to prove some useful 
properties about Ramsey numbers, but for the most part, it is difficult to find their exact 
values. See, for example, Radziszowski (2011); Lidicky and Pfender (2021), and Jaradat and 
Alzalg (2007, 2008), and the references contained therein. 


6.4 Permutations 
In order to find the number of possible arrangements of a set of objects, we use a concept 


called permutations. There are methods for calculating permutations, and it is important 
to understand the difference between a set with and without repetition. 
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6.4.1 Permutations Without Repetition 


Many counting problems can be resolved by determining the count of arrangements pos- 
sible for a specified number of distinct elements drawn from a set of a specific size, with 
an emphasis that the order of these elements matters. For example, in how many ways can 
we choose three singers from a group of five singers to perform three different songs in a 
concert, where each singer will perform exactly one song individually? In this section, we 
develop the present rules to solve counting problems such as this. 

To answer the question posed in the previous paragraph, note that the order in which we 
choose the singers matters. There are five ways to choose the first singer for the first song 
that will be performed at the start of the concert. Once this singer has been chosen, there 
are four ways to choose the second singer for the second song that will be performed in the 
middle of the concert. Once the first and second singers have been chosen, there are three 
ways to choose the third singer who will perform at the end of the concert. By the product 
rule, there are 5-4-3 = 60 ways to choose three singers from a group of five singers to 
perform three different songs in a concert, where each singer will perform exactly one song 
individually. 

A permutation ofa set of distinct objects is an ordered arrangement of some or all of these 
objects. Formally, we have the following definition. 


Definition 6.1 Let n and r be integers with 0 < r < n. An ordered arrangement of r ele- 
ments of a set with n distinct elements is called an r-permutation and is denoted by P(n, r). 


The product principle of counting can be used to find a formula for P(n, r). 


Theorem 6.10 Letn andr be integers with 0 < r < n. Then 


PG i Ge a 
(n—r)! 


Proof: We will employ the product rule to establish the correctness of this formula. Initially, 
there are n ways to select the first element of the permutation, given that there are precisely 
n elements in the set. Subsequently, for the second element of the permutation, there are 
n — 1 choices, as there remain n — 1 elements after utilizing the one picked for the first 
position. Following this pattern, the third element has n — 2 options, and so forth, until 
there are precisely n — (r — 1) = n — r + 1 ways to choose the rth element. Consequently, in 
accordance with the product rule, there are 


n! 
(n—r)! 


r-permutations of the set. This proves the theorem. a 


n(n—1)\n—-2)---(n-r+l1)= 


Let n be a nonnegative integer. Note that P(n, n) = n!. Note also that P(n, 0) = 1 because 
there is exactly one way to order zero elements. That is, there is exactly one list with no 
elements in it, namely, the empty set. 

We give some examples as direct applications of Theorem 6.10. 
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Example 6.15 In how many ways can we arrange all singers of a group of five singers 
to perform different songs in a concert, where each singer will perform exactly one song 
individually? 


Solution 
By Theorem 6.10, there are 


P(5,5) = 5!=5-4-3-2-1=120 


ways to select five singers from a group of five singers to perform different songs in a concert, 
where each singer will perform exactly one song individually. a 


Example 6.16 In how many ways can we form four distinct letter passwords from the 
letters: 
(a) A,B,C, D? (b) A, B, C, D, E, F? 


Solution 

(a) By Theorem 6.10, there are P(4,4) = 4-3-2-1 = 24 ways to form four distinct letter 
passwords from the letters A, B, C, D. 

(b) By Theorem 6.10, there are P(6,4) = 6- 5-4-3 = 360 ways to form four distinct letter 
passwords from the letters A, B, C, D, E, F. E 


6.4.2 Permutations with Repetition 


In many counting problems, elements may be used repeatedly. For instance, a letter may be 
used more than once on a password. Permutations when repetition of elements is allowed 
can be easily counted by using the product principle of counting. We have the following 
theorem. 


Theorem 6.11 The number of r-permutation of a set of n objects with repetition allowed 
is n”. 


Proof: When repetition is permitted in forming an r-permutation, there exist n options to 
pick an element from the set for each of the r positions. This is due to the fact that, for each 
selection, all n objects remain available for consideration. Consequently, by applying the 
product rule, it follows that there are a total of n” possible r-permutations when repetition 
is allowed. This proves the theorem. E 


The following examples are direct applications of Theorems 6.10 and 6.11. 


Example 6.17 


(a) In how many ways can 10 boys have 10 different birthdays? 
(b) In how many ways can 10 boys have 10 birthdays? 


Ignore the existence of the leap year, so a year has 365 days. 
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amin Buffalo E rmo) 


Figure 6.4 Daily round-trips between Columbus and Toronto, with connections in Cleveland and 
Buffalo. 


Solution 
(a) By Theorem 6.10, there are 


P(365, 10) = 365 - 364 - 363 - 362 - 361 - 360 - 359 - 358 - 357 - 356 


ways 10 boys have 10 different birthdays. 
(b) By Theorem 6.11, there are 365!° ways 10 boys have 10 birthdays. a 


Example 6.18 Suppose that there are three daily round-trip bus routes between Colum- 
bus and Cleveland, five daily round-trip bus routes between Cleveland and Buffalo, 
and four daily round-trip bus routes between Buffalo and Toronto. See Figure 6.4. A 
passenger would like to take a round-trip that departs from (and returns to) Columbus, and 
runs as follows: Columbus-—Cleveland—Buffalo-Toronto-—Buffalo—Cleveland—Columbus. 
In how many ways can the passenger take such a round trip in each of the following 
two cases? 


(a) The passenger cannot use the same bus more than once. 
(b) The passenger may use the same bus more than once. 


Solution 
(a) If the passenger cannot use the same bus more than once, then there are 
3-5-4-3-4-2=1440 
ways to take the planned round trip. This is a direct consequence of Theorem 6.10 since 
1440 = 3-(3-1)-5-(5-1)-4-(4-1). 
(b) If the passenger may use the same bus more than once, then there are 
3-5-4-4-5-3 = 3600 


ways to take the planned round trip. This is a direct consequence of Theorem 6.11 since 
3600 = 37. 5?- 4°. m 


6.5 Combinations 


In order to find the number of ways to select a particular number of elements from a set 
of a particular size, we use a concept called combinations. There are methods for calculat- 
ing combinations, and it is important to understand the difference between a set with and 
without repetition. 


6.5 Combinations 


6.5.1 Combinations Without Repetition 


Many counting problems can be resolved by determining the count of arrangements possi- 
ble for a specified number of distinct elements drawn from a set of a specific size, with an 
emphasis that the order of the elements selected does not matter. For example, how many 
different bands of three singers can be formed from a group of four singers? In this section, 
we develop present rules to solve counting problems such as this. 

To answer the question posed in the previous paragraph, note that the order in which 
we choose the singers does not matter. We need to find the number of subsets with three 
elements from the set containing the four singers. We see that there are four such subsets, 
one for each of the four singers, because choosing three singers is the same as choosing one 
of the four singers to leave out the group. This means that there are four ways to choose three 
singers for the band, where the order in which these singers are chosen does not matter. 

A combination of a set of distinct objects is an unordered arrangement of some or all of 
these objects. Formally, we have the following definition. 


Definition 6.2 Let n and r be integers with 0 < r <n. An unordered selection of r ele- 
ments from a set with n distinct elements is called an r-combination and is denoted by 
C(n, r). 


The product principle of counting can be used to find a formula for C(n, r). 


Theorem 6.12 Letn andr be integers with 0 < r < n. Then 
! 
C(n,r) = (") s 
7 


(n—r)!r!" 


Proof: The r-permutations, P(n,r), for the set can be derived by first forming the r- 
combinations, C(n, r), and then arranging the elements within each r-combination. The 
arrangement of elements within an r-combination can be done in exactly P(r,r) ways. 
Consequently, by applying the product rule, it follows that P(n, r) = C(n,r) - P(r,r). This 
relationship implies the desired result that 

P(n,r) _ n!/(n-r)! B n! 


On amo ee a-ri m 


Returning to the question posed at the beginning of this section, we find that we can 
form C(4, 3) = a = 4 different bands of three singers from a group of four singers. As we 
mentioned earlier, choosing three singers is the same as choosing one of the four singers 
to leave out the group. That is, C(4, 3) = C(4, 1) a 4. More generally, we have the 
following corollary. 


Corollary 6.3 Letn andr be integers with 0 < r < n. Then C(n,r) = C(n,n — r). 


Proof: The result follows immediately from Theorem 6.12 and the identity given 
in (6.1). E 


The following examples are direct applications of Theorem 6.12. 
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Example 6.19 How many pairs can be chosen from a group of six people? 


Solution 
By Theorem 6.12, we can choose C(6, 2) = e ) = 15 pairs from a group of six people. m 


Example 6.20 Suppose that a class consists of six boys and four girls. 


(a) How many ways are there to choose a group of five students? 
(b) How many ways are there to choose a group of five students if the group consists of 
three boys and two girls? 


Solution 
(a) By Theorem 6.12, the number of ways to select the group is 


C(10, 5) = (3) = 252. 


(b) By the product principle of counting, the answer is the product of the number of 3- 
combinations of boys and the 2-combinations of girls. By Theorem 6.12, the number of 
ways to select the group is 


C(6, 3) - C(4,2) = (3) (5) = (20)(6) = 120. 


This answers the questions raised in the example. a 


6.5.2 Combinations with Repetition 


Let us say there are five flavors of ice cream: banana (B), chocolate (C), lemon (L), straw- 
berry (S), and vanilla (V). We can have three scoops. Example selections include: 


e {C, C, C}, which means that three scoops of chocolate were selected. 
e {B, L, V}, which means that one each of banana, lemon, and vanilla was selected. 
e {B, V, V}, which means that one of banana and two of vanilla were selected. 


We are interested in how many variations there will be. This is an example of a combi- 
nation with repetition of elements allowed. Note that there are five things to choose from, 
and we choose three of them. Also note that the order in which the scoops are selected does 
not matter, and we can repeat! Therefore, this example involves counting 3-combinations 
with repetition allowed from a 5-element set. Below we show a technique for solving this 
counting problem, which leads us to a general method for counting the r-combinations 
with repetition allowed from an n-element set. 

Let us think about the ice cream problem posed above as ice cream being in boxes (see 
Figure 6.5) and a robot being ordered remotely to get the desired selections. For instance: 


e To select {C, C, C}, we send the orders “move past the first box, then take three scoops, 
then move along three more boxes to the end.” In other words, we send the orders “move 
once, then take three scoops, then move thrice to the right.” 


TEA fs |v 


Figure 6.5 Five different kinds of ice cream in five boxes (containers). 


6.5 Combinations 


e To select {B, L, V}, we send the orders “take a scoop, then move twice to the right, then 
take a scoop, then move twice to the right, then take a scoop.” 

e To select {B, V, V}, we send the orders “take a scoop, then move quarce to the right, then 
take two scoops.” 


“oD 


Let us write “=>” to mean that “move once to the right,” and “o” to mean that “take one 
scoop.” The above three selections can be written as follows: 


e The selection {C, C, C} is represented as “> ooo > > >.” 
e The selection {B, L, V} is represented as “o > > o > > o.” 
e The selection {B, V, V} is represented as “o > > > > o o.” 


From the above discussion, the above counting question can be simplified as follows: 
How many different ways can we arrange arrows and circles? 

Note that there are always three circles, so we always have three scoops of ice cream, 
and there are always four arrows because we need to move four times to go from the first 
container to the fifth container. Therefore, there are seven positions, and we want to choose 
three of them to have circles. That is, we have 3 + (5 — 1) positions and want to choose 3 of 
them. So, the number of ways of having three scoops from five flavors of ice cream is 


RA e as 
3 3 1-2-3 


We can also look at the arrows instead of the circles, and say that we have 3 + (5 — 1) posi- 
tions and want to choose (5 — 1) of them to have arrows. The following theorem generalizes 
this discussion. 


Theorem 6.13 There are 
Cintr-1n-1=Cntr-1n= ("rN") 
E 


r-combinations from a set with n-elements when repetition of elements is allowed. 


Proof: To represent each r-combination from a set with n elements when repetition is 
allowed, a combination can be depicted using n — 1 right arrows and r circles. These n — 1 
right arrows are employed to separate n distinct cells, where each cell corresponds to an 
element in the set, and the ith cell contains a circle for each occurrence of the ith element 
within the combination. For instance, consider a 5-combination from a set of four elements, 
which is represented as three arrows and five circles: 


OOO =F OO 


This representation signifies a combination comprising exactly two occurrences of the first 
element, none of the second element, one of the third element, and two of the fourth ele- 
ment from the set. 

As demonstrated, each unique list consisting of n — 1 arrows and r circles corresponds to 
an r-combination derived from the set with n elements, permitting repetition. The count of 
such lists can be computed as C(n — 1 + r, r), as each list corresponds to selecting r positions 
out of the n — 1 + r positions that encompass both circles and arrows. This count also equals 
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C(n —1+1r,n—1) because each list corresponds to choosing n — 1 positions to place the 
n — 1 arrows. E 


The following examples show how Theorem 6.13 is applied. 


Example 6.21 Suppose there are four varieties of donuts: Chocolate (C), Glazed (G), 
Pumpkin (P), and Raspberry (R). 


(a) Find the number of ways one can select 30 donuts. 
(b) Find the number of ways one can select 30 donuts such that the selection includes at 
least 2 C donuts, 2 G donuts, 3 P donuts, and 4 R donuts. 


Solution 

Note that the order in which the donuts can be selected does not matter, and the donuts can 
be repeated. The total number of varieties of donuts is n = 4. The number of donuts to be 
selected is r = 30. 


(a) This count would be the number of 30-combinations with repetition allowed from a 
set with six elements. From Theorem 6.13, the donuts can be selected in the following 
different ways: 


C(4+ 30 — 1, 30) = C(33, 30) = 232-31 Z 5456, 
1-2-3 
(b) To satisfy the requirements, we preselect the minimum number of each type: 2 C, 2 G, 
3 P, 4R. We then have at most 30 — 2 — 2 — 3 — 4 = 19 donuts left to choose from, now 


without any restrictions. This count would be 


C44 194,10) = C(22,19) = 22—21 cay, 
1-2-3 
This answers the questions raised in the Example 6.21. E 


Example 6.22 How many solutions does the equation x, +x, +X, +x, = 13 have, where 
X1, X2, X3, and x, are nonnegative integers? 


Solution 
To determine the count of solutions, we can observe that a solution corresponds to a manner 
of selecting 13 items from a set containing four distinct elements, with x, items of type one, 
x, items of type two, x, items of type three, and x, items of type four. In this context, the 
number of solutions equates to the count of 13-combinations with repetition allowed from 
a set comprising four elements. Applying Theorem 6.13, we conclude that there are 

C(4 +13 — 1,13) = C(16, 13) = C(16, 3) = Lo = 560 


possible solutions. E 


Example 6.23 What is the value of “sum” after the fragment given in Algorithm 6.4, 
where n, m, p, and q are positive integers, has been executed? 


6.5 Combinations 


Algorithm 6.4: The algorithm of Example 6.23 
1: sum = 0 

2: for (i = 1;i < n;i + +) do 

3: | for (j= 1;j < i;j++)do 

4: for (k = 1;k < j;k + +) do 

5 for (m = 1;m < k; m + +) do 

6 | sum = sum + 1 

7: end 

end 

9: | end 


10: end 


Solution 
Note that the initial value of “sum” is 0 and that 1 is added to “sum” each time the nested 
loop is traversed with a sequence of four letters i, j, k, and r such that 


l<m<k<j<i<n. 


The number of such sequences of integers is the number of ways to choose four integers 


from {1, 2, ..., n}, with repetition allowed. 
Hence, from Theorem 6.13, it follows that 
sum = C(n +4- 1,4) = Grae = s(n! + 6n? + 11n? + 6n) 


after this code has been executed. E 


The formulas for the numbers of ordered and unordered selections of r elements, 
chosen with and within repetition allowed from a set with n elements, are shown in 
Table 6.2. 


Table 6.2 Permutations and combinations with and without repetition. 


Without repetition With repetition 
i n! 5 
r-permutations m n” 
(n—r)!r! 
ete n! (n+r—1)! 
r-combinations —— —_— 
(n—r)!r! (n—1)!r! 


6.5.2.1 Using Combinations to Find Permutations with Indistinguishable Objects 

Some elements may be indistinguishable in counting problems. When this is the case, care 
must be taken to avoid counting things more than once. In particular, we can use combina- 
tions to find the number of different permutations with indistinguishable objects. We have 
the following theorem. 
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Theorem 6.14 The number of different permutations of n objects, where there are n, 
indistinguishable objects of type 1, n, indistinguishable objects of type 2, ... , ną indistin- 
guishable objects of type k, is 
n! 
n!m! +--+ ny! 


Proof: To calculate the count of permutations, we can start by observing that the n, objects 
of type 1 can be arranged among the n positions in C(n, n,) ways, which leaves n — n, posi- 
tions unoccupied. Following this, the objects of type 2 can be positioned in C(n — n,,n,) 
ways, resulting in n — n; — n, available positions. This process continues for objects of type 
3, and so on, up to type k — 1, until finally, n, objects of type k can be accommodated in 
C(n—n, — N, — : +: — Ng» Ng) ways. Therefore, by applying the product principle of count- 
ing, the total number of distinct permutations can be calculated as follows: 


C(n,n,)C(n — ni, ng): :-C(n — n — n, — +: — Nging) 
n! (n—n,)! (n-n; Ny_,)! n! 
~ nı!(n- n)! nm !(n -n —n,)! n,!0! n!n!: en! 
The proof is complete. E 


The following example shows how Theorem 6.14 is applied. 


Example 6.24 Determine the number of ways to arrange six letters of the word BANANA. 


Solution 
We count permutations of three A’s, two N’s, and one B, a total of six symbols. By 
Theorem 6.14, the number of these is < = 60. E 


6.5.3 Distributing Objects into Distinguishable Boxes 


Many counting problems can be solved by enumerating the ways objects, distinguishable 
(i.e., different) or indistinguishable (i.e., identical), can be placed into boxes that are distin- 
guishable (often called labeled) or indistinguishable (often called unlabeled). 

There are closed formulas for counting the ways to distribute objects, distinguishable or 
indistinguishable, into distinguishable boxes, but there are no closed formulas for counting 
the ways to distribute objects, distinguishable or indistinguishable, into indistinguishable 
boxes. 


6.5.3.1 Distributing Distinguishable Objects into Distinguishable Boxes 
Counting problems that involve distributing distinguishable objects into distinguishable 
boxes can be solved using the following theorem. 


Theorem 6.15 The number of ways to distribute n distinguishable objects into k distin- 
guishable boxes so that n; objects are placed into box i, i = 1, 2, ... ,k, equals 
n! 


Exercises 


The proof of Theorem 6.15 uses the product principle, similar to the proof of 
Theorem 6.14, and it is therefore omitted. It can also be proved by setting up a one- 
to-one correspondence between the permutations counted in Theorem 6.14 and the ways 
to distribute distinguishable objects counted by Theorem 6.15. The following example is 
taken from (Rosen, 2002, Section 6.5). 


Example 6.25 How many waysare there to distribute hands of 5 cards to each of 4 players 
from a card game of 52 cards?° 


Solution 

This counts the number of ways to distribute 52 distinguishable objects into 5 distinguish- 
able boxes so that 5 objects are placed into the first box, 5 objects are placed into the second 
box, 5 objects are placed into the third box, 5 objects are placed into the fourth box, and 
32 objects (which represent 52 — 4 x 5 remaining cards) are placed into the fifth box. By 
Theorem 6.15, these are 52!/(5!5!5!5!32!) ways. a 


6.5.3.2 Distributing Indistinguishable Objects into Distinguishable Boxes 
Counting problems that involve distributing indistinguishable objects into distinguishable 
boxes can be solved using the following theorem. 


Theorem 6.16 There are 
Cn+r-1,n-1)=C(n+r-1,r)= (rtr) 
r 


ways to place r indistinguishable objects into n distinguishable boxes. 


Theorem 6.16 is similar to the proof Theorem 6.13 and it is therefore omitted. It can also 
be proved by setting up a one-to-one correspondence between the combinations counted in 
Theorem 6.13 and the ways to place distinguishable objects counted by Theorem 6.16. 


Example 6.26 How many ways to place 12 indistinguishable homemade doughnuts into 
9 distinguishable plates? 


Solution 
This counts the number of ways to place 12 indistinguishable objects into 9 distinguishable 
boxes. By Theorem 6.16, these are 


C9 +12 -1,9 — 1) = C(20,8) = K = 125,970 


ways. E 


Exercises 


6.1 Choose the correct answer for each of the following multiple-choice questions/ 
items. 
(a) The coefficient of xy!” in the expression (2x — 3y)” is 


6 The standard deck consists of 52 cards. 
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: 25! 442413 i 25! 419413 

1) — 2s Ds 1i 2 : 

@) 12!13! ay 12!13! 

ea 25! 13512 * 25! 13412 
Z iv) ——2°3 

GD -gma 3 OY) erat 


(b) There are cats in three rooms. The first room has four cats, the second room has 
five cats, and the third room has three cats. In how many ways to choose a cat 
from these three rooms? 

(i) 12. (ii) 24. (iii) 48. (iv) 60. 

(c) Let A and B be two disjoint sets. If |A| = 3|B| = 6, then the cardinality of the 
powerset of A x B is 
(i) 8. (ii) 12. (iii) 256. (iv) 4096. 

(d) In an exam, there are seven true/false questions and eight multiple-choice ques- 
tions for which the answers can be (i), (ii), (iii), (iv). The number of different ways 
of answering the exam is 


(i) 27x47. (ii) 27 x4. (iii) 28 x4. (iv) 28x 48, 
(e) How many even 4 digit whole numbers are there? 
(i) 256. (ii) 625. (iii) 4500. (iv) 5000. 


(f) A professor gives a multiple-choice quiz that has eight questions, each with four 
possible answers (i), (ii), (iii), (iv). What is the minimum number of students that 
must be in the professor’s class in order to guarantee that at least four answer 
sheets must be identical? (Assume that no answers are left blank.) 

(i) 48. (ii) 48 +1. (iii) 2x 48 41. (iv) 3x48 +1. 

(g) How many ways can we assign four problems to four students to solve them so 

that each student solves one problem? 


(i) 12. (ii) 16. (iii) 24. (iv) 48. 
(h) How many numbers of two digits can be formed with digits 1, 3, 5, 7, and 9? 
(i) 60. (ii) 120. (iii) 180. (iv) 240. 
(i) What is the number of diagonals can be drawn in an octagon? 
(i) 20. (ii) 28. (iii) 34. (iv) 42. 
(j) Let A, Band C be three disjoint sets. If |A| = |B] + 1 = |C] + 2 = 4, then the cor- 
diality of A U Bu C is 
(i) 9. (ii) 24. (iii) 512. (iv) 4096. 
(k) The number of subsets of a 99-element set is 
(i) 2°. i) e ). (iii) 99. (iv) i ). 
(1) The number of 9-element subsets of a 99-element set is 
ya: Gi) ( . j. (iii) 99. (iv) ( ) 


(m) The number of 7-letter upper-case words is 


(i) 267. (ii) 257. (iii) 26” — 257. (iv) E 
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(n) The number of 7-letter upper-case words that do not contain the letter B is 
(i) 267. (ii) 257. (iii) 267 — 257. (iv) . i 

(o) The number of 7-letter upper-case words that contain the letter B is 
(i) 267. (ii) 257. (iii) 26” — 257. a t ). 

(p) The number of 7-letter upper-case words whose letters are distinct and occur in 
alphabetically increasing order is’ 


(i) 267. (ii) 257. (iii) 26” — 257. (iv) T i: 
6.2 Use the binomial theorem to find the expansion of (x + y)°. 
6.3 Give a combinatorial proof for item (a) of Corollary 6.1. 
6.4 Prove item (d) of Corollary 6.1. 
6.5 Give a combinatorial proof for Pascal’s identity (Theorem 6.2). 
6.6 Give a combinatorial proof for Vandermonde’s identity (Theorem 6.3). 
6.7 Use mathematical induction to prove Theorem 6.4. 
6.8 Use mathematical induction to prove Theorem 6.5. 
6.9 Use contradiction to prove Theorem 6.8. 


6.10 Let A, B, and C be three sets such that |B N C| = 19 and [A N BN C| = 11. Find |A’ N 
BNC|. 


6.11 A survey of 240 people showed that 91 like tea, 70 like coffee, 31 like tea and coffee, 
91 like neither coffee nor tea, and in addition do not like milk, and 7 like coffee, 
tea, and milk. How many like milk only? Justify your answer. (Hint: Use a Venn 
diagram). 


6.12 What is the value of “sum” after the fragment given in Algorithm 6.5, where n, m, 
p, and q are positive integers, has been executed? 


6.13 Prove that a function f from a set with k + 1 or more elements to a set with k elements 
is not an injection. 


6.14 Zaid has six different colored shirts. In how many ways can he hang the four shirts 
in the cupboard? 


7 For example, ABEL is counted, but not ABLE (since L and E are not in alphabetical order), nor APPEL 
(since P is repeated). 
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Algorithm 6.5: The algorithm of Exercise 6.12 


1: sum = 1 

2: for (i = 1;i < n;i + +) do 

3: | for (j = 0;j < m;j++)do 

4 sum = sum + 1 

5: for (k = p;k > 1; k — —) do 
6: | sum = sum + 1 

7: end 

8: for (r = q;r > 0;r — —) do 
9: | sum = sum + 1 

10: end 

u: | end 

12: end 


6.15 Determine the number of ways to arrange: 
(a) The seven letters of the word ONEWORD. (Note that this word has two O’s, one 
N, one E, one W, one R, and one D:) 
(b) The word ONEWORD if the two O’s must be adjacent (as in NEWDOOR). 


6.16 Determine the number of ways to arrange: 
(a) The nine letters of the word PINEAPPLE. (Note that this word has three P’s, two 
E’s, one I, one N, one A, and one L:) 
(b) The nine letters of the word PINEAPPLE if the three P’s must be adjacent (as in 
APPPLENIE). 


Notes and Sources 


The history of counting is as ancient as human civilization itself. Counting can be traced 
back to prehistoric times when early humans used simple tally marks, notches, or pebbles 
to keep track of quantities; see Ifrah (2000). As societies developed, more advanced systems 
of counting and numbering emerged. One of the earliest known numerical systems is the 
Babylonian cuneiform script, dating back to around 2000 BCE, which used a base-60 (sex- 
agesimal) numbering system. The ancient Egyptians also developed their numeral system, 
hieroglyphs for numbers, and the concept of place value. The Indian subcontinent played 
a significant role in the history of counting, with the introduction of the decimal system 
and the concept of zero, which led to the development of modern numeral systems (see, for 
instance, Smith and Karpinski (2013)). In particular, the Arabic numeral system, with its 
ingenious use of 10 symbols and the concept of 0, has not only revolutionized mathematics 
but also become the foundation for modern global numerical representation, providing an 
efficient and universally adopted system for counting. 

This chapter provided the reader with the essential tools and knowledge required for 
counting and effective enumeration. It offered a comprehensive exploration of fundamental 
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concepts in counting. More precisely, we introduced binomial coefficients and their manip- 
ulation and identities. Then, we delved into the principles of counting, gaining insight into 
various counting techniques, including permutations and combinations. 

As we conclude this chapter, it is worth noting that the cited references and others, such 
as Cusack and Santos (2021); Mott et al. (1986); Liu (1968); Grimaldi (1999); Yellen et al. 
(1999); Fleming and Pitassi (2021); Beame and Riis (1996), and Knuth (1997), also serve as 
valuable sources of information pertaining to the subject matter covered in this chapter. The 
code that created Figure 6.1 is due to StackExchange (2011). The code that created the right- 
hand side picture in Figure 6.2 was taken from the source file of Alzalg and Alioui (2022). 
The source code of the Jordan map in Figure 6.2 is publicly available at http://sites.ju.edu 
.jo/sites/alzalg/pages/jordanmap.aspx. The code that created Figure 6.3 is due to StackEx- 
change (2016). 
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An algorithm is a finite set of precise instructions that are performed to solve a problem. 
So, every algorithm is constructed using a finite sequence of statements. The term “algo- 
rithm” has its origins in the work of Muhammad Ibn Musa Al-Khwarizmi (ca. 780-840). 
His book, which focused on the Arabic numeral system and various computational methods 
such as addition, subtraction, multiplication, division, handling fractions, and calculating 
roots, played a groundbreaking role in the development of modern mathematics. This work 
by Al-Khwarizmi had a profound impact on the field (refer to Khuwarizmi- et al. (1963)). 

In the first half of this chapter, we present the fundamentals of runtime analysis, and 
study the asymptotic analysis of algorithms. In the second half of this chapter, we analyze 
sequential programs, which are programs without function calls. We also analyze recursive 
and nonrecursive programs, which are programs with function calls. This chapter ends by 
briefly introducing the complexity classes NP and NP-complete. 

In mathematics, asymptotic analysis is a method of describing limiting behavior. 
For instance, the study of the properties of a function f(x) as x becomes very large. 


As an example, the graph of the function f(x) = 5 approaches 1 as x goes to infinity. 
See Figure 7.1. 
In fact, 
lim ~~ = lim —~— = lim — =: 


xe0x — 2 x>% x(] — 2) x01 _ 2 
x x 


Therefore, mathematically speaking, we call the line y = 1 a horizontal asymptotic for f(x). 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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ty Figure 7.1 The graph of f(x) = ST 
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In computer science, asymptotic analysis involves assessing the efficiency of an algorithm 
concerning the input size, denoted as n, particularly when n becomes very large. What is 
an algorithm? How can we construct an algorithm for performing certain tasks? And how 
can we compare between various algorithms for solving a certain problem? This will be 
addressed in the first section. 


7.1 Constructing and Comparing Algorithms 


In the first part of this section, we present some decision-making statements that are used 
to build blocks of algorithms, and introduce the running time of an algorithm. 


7.1.1 Basic Tools for Constructing Algorithms 


Decision-making statements can be placed into the following categories: 


7.1.1.1 Simple Statements 
The following are simple statements in the programming language C. 


e Expressions: This includes 
- printf; (write statement - print formatted data). 
- scanf; (read statement — read formatted data). 
- assignment statements (such as “set a=1;”). 
e Jump statements, such as goto; break; continue; return; etc. 
e The null statement: This contains only a semicolon “;”. It is known that nothing happens 
when a null statement is executed. 


We note that, in the programming languages such as C, simple statements end in a 
semicolon. 


7.1 Constructing and Comparing Algorithms 


7.1.1.2 If-Statement 
The “if-statement” and “if-else-statement” are also called selection statements. In 
Algorithm 7.1, we present a simple code for “if-statement.” Here statement(s) is (are) 
executed only if condition is true. See Figure 7.2 (a). 

In Algorithm 7.2, we present a simple code for “if-else-statement,” where S, denotes a set 
of statement(s), and S, denotes another set of statement(s). Here, if condition is true, then 
S, is executed. Otherwise, S, is executed. See Figure 7.2 (b). 


Example 7.1 In Algorithm 7.3, we present a code for finding the largest of two integers. 
a 


Algorithm 7.1: Writing an if-statement 


1: if some condition is true then 
2: | do some statement(s) S 
3: end 


Algorithm 7.2: Writing an if-else-statement 


1: if some condition is true then 

2: | do some statement(s) S, 

3: end 

4: else 

5: | do some different statements S, 
6: end 


Algorithm 7.3: Finding the largest of two integers 

Input: Integers a, b 
Output: Bigger value of a, b 

1: if (a > b) then 

2: | print f£(“bigger value = ”a) 

3: end 

4: else 

5: | print f(“bigger value = ”b) 

6: end 


7.1.1.3 For-Statement 
Algorithm 7.4 shows a simple code that represents “for-statement.” See also Figure 7.2 (e). 


Algorithm 7.4: Writing a for-statement 


1: for (initialize; test; in/decrement) do 
2: | do some statement(s) 
3: end 


Example 7.2 Algorithm 7.5 presents a code for finding the maximum of integers. E 


183 


184] 7 Analysis of Algorithms 


else-part 


(a) (b) 


od 


(c) (d) 


reinitialize 


G) 
— (e) " OnO 


(e) 


Figure 7.2 Flowcharts showing basic statements. (a) If-statement in a flowchart, (b) If-else in a 
flowchart, (c) While loop in a flowchart, (d) Do-while loop in a flowchart, (e) For-loop in a flowchart, 
and A block in a flowchart. 
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Algorithm 7.5: Finding the maximum of integers 


Input: Integers a,,a,,...,a, 
Output: Largest value of a,,a,,...,a, 
1: Max = a, 


2: for (i = 0;i < n; i + +) do 

3: | if (max <a,) then 

4 | max = a; 

5 | end 

6: end 

7: print f£(“largest value = ”max) 


7.1.1.4 While-Statement 
In Algorithm 7.6, we present a simple code for “while statement.” Here the body 
(statement(s)) is executed as long as condition is true. See Figure 7.2 (c). 


Algorithm 7.6: Writing a while-statement 
1: while some condition is true do 
2: | do some statement(s) 
3: end 


Example 7.3 Algorithm 7.7 contains a while loop that prints the numbers 1, 2, ...,5. m 


Algorithm 7.7: Using a while-statement to print numbers 1-5 
Input: Integer i 
Output: Print numbers from 1 to 5 
1: inti=1 
2: while (i < 5) do 
3: | printf@ // Output is 1 2 3 4 
4: ++i 
5: end 


vI 


6: return 0 


7.1.1.5 Do-While-Statement 
In Algorithm 7.8, we present a simple code for “do-while statement.” Note that the do- 
while is similar to the while-statement except that the body (statement(s)) is executed at 
least once. See Figure 7.2 (d). 


Algorithm 7.8: Writing a do-while-statement 
1: do 
2: | do some statement(s) 
3: while some condition is true 
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Example 7.4 Algorithm 7.9 contains a do-while loop that writes the numbers 1, 2, ... , 5. 
fs 


Algorithm 7.9: Using a do-while-statement to print numbers from 1 to 5 
Input: Integer i 
Output: Print numbers from 1 to 5 
1: inti=1 
2: do 
3: | printf(i) // Outputis12345 
4: ++i 
s: while (i < 5) 
6: return 0 


7.1.1.6 Block 
If S1, S2, ...,S„ are statements, then the code in Algorithm 7.10 is called a block. See 
Figure 7.2 (f). 


Algorithm 7.10: Writing a block 


1: do a statement S, 
2: do a statement S, 
3 i 

4: do a statement S, 


The following example shows a block with three statements. 


Example 7.5 In Algorithm 7.11, we present a code that writes the letters a,b, and c in 
three lines. Here, the command “\n” means jumping to a newline. E 


Algorithm 7.11: Printing the letters a, b, and c 


1: printf(“a\n”) // Output is a 
2: print f£(“b\n”) // Output is b 
3: print f(“c”) // Output is c 


The following algorithm example is due to Kuchling (2012). 


Example 7.6 A straightforward algorithm named £ind-max has been devised for the 
following problem: 

Prose: Given an array of positive numbers, return the largest number of the array. 
Input: An array A of positive numbers. This list must contain at least one number. (Asking 
for the largest number in a list of no numbers is not a meaningful question.) 

Output: A number that will be the largest number of the list. 


(a) Construct an algorithmic code for the given procedure as follows: 
1. Set max to 0. 
2. For each x in the list L, compare it to max. If x is larger, set max to x. 
3. max is now set to the largest number in the list. 
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(b) Examine whether the algorithm formulated in item (a) possesses well-defined inputs 
and outputs, ensures termination, and yields accurate results. 


Solution 
(a) Acode that solves the given problem is stated in Algorithm 7.12. 


Algorithm 7.12: Finding the largest number of an array: find-max(A, n) 
Input: An array A[0 : n — 1] of positive integers and length n 
Output: The largest number of the array 
1: max = 0 
: for (i = 0;i < n;i + +) do 
if (A[i] > max) then 
| max = A[i] 


Bw 


5 | end 
6: end 
7: return max 


(b) The algorithm in item (a) has well-defined inputs and outputs, guarantees termination, 
and produces accurate results. Note that the input for Algorithm 7.12 is a finite array of 
positive numbers whose length is n, and the output is the maximum in the array. Note 
also that Algorithm 7.12 terminates after visiting all n elements, and that the “for” loop 
from line (2) to line (6) searches through the array and assigns any value larger than 
max to max. | 


It is not in the scope of this book to provide the reader a comprehensive list of the control 
statements because this is an asymptotic analytically oriented book chapter. Such a list can 
be found in any book about C programming. The switch statement, for instance, is known 
as one of the selection statements available in computer programming languages. 


7.1.2 Choosing and Comparing Algorithms 


Some problems can be solved by more than one algorithm. How, then, should we choose 
an algorithm to solve a given problem? If we are tasked with developing a program for a 
one-time use with small data sets, it is advisable to opt for the simplest algorithm avail- 
able. However, in scenarios where the program is intended for long-term use and will be 
maintained by multiple individuals, additional factors come into play: 


e Simplicity: An algorithm that is straightforward and easy to understand is more likely to 
be implemented correctly compared to a complex one. 

e Clarity: Well-written, well-documented algorithms are more manageable for others to 
maintain over time. 

e Efficiency: Efficiency becomes crucial as the problem size increases. While efficiency 
often relates to the runtime of a program, it can also encompass other resource con- 
siderations, such as storage space. For larger problems, the runtime is a primary factor 
in choosing an algorithm. We shall, in fact, take the efficiency of an algorithm to be its 
running time. 
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Algorithm 7.13: Fragment A of Example 7.7 


1: array[0] = 0 // Cost is c} 
2: array[1] = 0 // Cost is c, 
3: array[2] = 0 // Cost is c, 
4.3 

5: array[n — 1] = 0 // Cost is c, 


// Total cost isc, +c; +++: +c, = nc 


Algorithm 7.14: Fragment B of Example 7.7 


1: for (i = 0;i < n;i + +) do // Cost is cy 
2: | array[i] = 0 // Cost is c, 
3: end 


// Total cost is (n + 1)cy + nc, = (Co + c)n + Cy 


Definition 7.1 The running time of an algorithm is the duration it requires to complete 
a task, quantified as a function of the input size. 


Examples of input size (number of elements in the input) are: 


e Size of an array. e Degree of a polynomial. 
e Vertices and edges of a graph. e Number of elements in a matrix. 


Given n, the size of the input, we can express the running time as a function of the input, 
say f(n). 


Example 7.7 For each of the two fragments in Algorithms 7.13 and 7.14, we associate a 
“cost” with each statement as follows: cy is the cost of checking the for-statement state- 
ment’s condition at every iteration in Fragment B shown in Algorithm 7.14, and c; is the 
cost of executing the assignment statement array[i] = 0 for each i = 0,1, ...,n — 1. 

We find the “total cost” by finding the total number of times each statement is executed. 
The total cost of performing Fragment A shown in Algorithm 7.13 is f(n) = cn. 

The body of the “for” statement is executed n times (from 0 to n- 1). The “for” statement 
is executed n times (from 0 to n-1), plus 1 for the last time that i is checked and the 
inequality is false, that is, n+1 times. Thus, the total cost of performing Fragment B shown 
in Algorithm 7.14 is g(n) = (n + 1)cọ + ne, = (Cy +C))N + Cy. 

Now, when cy > 0, can we say that Algorithm 7.13 is more efficient than Algorithm 7.14? 
Note that both fragments have linear cost-time. We may find an answer to this question at 
the end of Example 7.8. E 


In general, to compare two algorithms, we compare the running time function for each 
algorithm. We will see a rough measure that characterizes how fast function grows, that is, 
the rate of growth. We compare functions for large values of n, that is, we compare functions 
asymptotically (in the limit). 
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Example 7.8 Suppose that for some problem we have the choice of using a linear-time 
program whose running time is f(n) = 50n and a quadratic-time program whose running 
time is g(n) = n?. The graphs of the running times are shown in Figure 7.3. 

The natural question that arises now is which algorithm is faster? Indeed, we are always 
interested in which is fastest asymptotically as n gets very large (we can think of n as going 
to infinity). So, clearly, the linear-time program is faster than the quadratic-time program. 

Another question that also arises is what impact do the constants have on the running 
time? Asymptotically speaking, it is clear that the constants have little or no impact on the 
running time. This might answer the question that we had at the end of Example 7.7. m 


7.2 Running Time of Algorithms 


In Section 7.1.1, we defined the running time of an algorithm to the amount of time it takes, 
measured as a function of the size of its input, such as the number of elements in a matrix, 
number of vertices and edges of a graph, etc. We also took the efficiency of an algorithm 
to be its running time. Given n, the size of the input, we can express the running time as a 
function of the input, f(n). 

In this section, we study line-by-line runtime analysis of algorithmic fragments, present 
types of runtime analysis of algorithms, determine upper/lower bounds for running time, 
and provide more examples on running time. 


7.2.1 Line-by-Line Runtime Analysis 


The runtime analysis that we did in Example 7.7 is called a line-by-line analysis. In this 
kind of runtime analysis, we multiply the cost of execution of each line by the number of 
times we execute that line and sum the results of each line. In this part, we present more 
examples on performing line-by-line runtime analysis. 
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Algorithm 7.15: The algorithm of Example 7.9 


1: sum = 0 // Cost = c}, # times = 1 
2: for (i = 0;i < n;i + +) do // Cost = c,, # times =n +1 
3: | for (j= 0;j <n;j++)do // Cost = c,, # times = n(n + 1) 
4: | sum += array[i][j] // Cost = c}, # times = n? 
5 | end 

6: end 

Example 7.9 Find the “total cost” needed to perform the fragment given in 


Algorithm 7.15 by finding the total number of times each statement is executed. 


Solution 

Let us add comments showing the number of times each statement is executed (see the side 
comments in gray in Algorithm 7.15). The following points are noted in order to find the 
total running time: 


e Itis clear that the assignment statement in line (1) is executed once. 

e How many times the “for” statement in line (2) executes? 0 to n — 1, which is n times, 
plus 1 for the last time that i is checked and the inequality is false, that is, n + 1 times. 

e How many times the “for” statement in line (3) executes? If the “for” statement in line (3) 
was not nested it would execute n + 1 times just like the “for” statement in line (2). Since 
the “for” statement in line (3) is nested it executes n + 1 times for each time we execute 
the body of the “for” statement in line (2) which is n times. 

e What about the sum statement in line (4)? If the second loop was not nested, the sum 
statement in line (4) would execute n times since that is the number of times the body of 
the second loop would execute. Since the second loop is nested, the sum statement in line 
(4) executes n times for each time we execute the body of the first loop which is n times. 
Therefore, n x n = n?. 


Now, the total cost or running time of the algorithm is the sum of the cost multiplied by 
the number of times for each line. 

Running time isc,(1) +c,(n + 1) + c,n(n + 1) + c,n”. Simplifying to polynomial form, we 
conclude that the running time is 


(c, + c3)n? + 2c,n + (C1 + c3), 


or cn? + ĉn + Č, where € = c, + C3, ĉ = 20,3, Č = C) + C2. 
We use the predominate term or in other words, the degree of the polynomial, to express 
the running time. So, we conclude the running time is cn, where c = c, + c3. a 


In Example 7.9, we found that the condition in the for-statement “for (i = 0; 
i < n;i + +)” was checked n + 1 times. Similarly, the condition in each of the for-statements 
“for (i = n;i > 0;i— —),” “for (i=1;i<=n;i++),” and “for (i= n;i >= 1;i—-—)” is 
checked n + 1 times. We have the following example. 


Example 7.10 Find the running time of the fragment given in Algorithm 7.16 by finding 
the total number of times each statement is executed. 
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Algorithm 7.16: The algorithm of Example 7.10 


1: intq =0 // Cost = c}, # times = 1 
2: for (i = 1;i < n;i + +) do // Cost = c, # times =n + 1 
3: | g=qt? // Cost = c}, # times = n 
4: end 

5: for (j =n; j = 1;j — —) do // Cost = c, # times =n + 1 
6: | q=q+j // Cost = c,, # times = n 
7: end 

Solution 


Note that this algorithm comprises two independent loops, each performing slightly distinct 
tasks. We have added side comments in gray showing the number of times each statement 
is executed. 

The running time of the fragment is the sum of the cost multiplied by the number of times 
for each line. Therefore, 


Running time = c} + c,(n + 1) + cn + c,(n + 1) + cn 
= (2c, + C3 + ¢4)n + (c) + 2c,) = CN + Ĉ, 


where c = 2c, +c, +c, and ¢ = c; + 2c,. Using the predominate term, the running time 
is cn. E 

In Example 7.10, we found that the condition in the for-statement “for (i= 1; 
i <= n;i + +)” is checked n+ 1 times. One can also find that the condition in each 
of the for-statements “for (i = 1; i <= logn; i + +),” “for (i = 1; i < n; i*=2),” and “for 
(i= n; i> 1; i/ = 2)” is checked (logn) + 1 times. 


Example 7.11 Find the running time of the fragment given in Algorithm 7.17 by finding 
the total number of times each statement is executed. 


Algorithm 7.17: The algorithm of Example 7.11 


1: intq=0 // Cost = c}, # times = 1 
2: for (i = 1;i < logn; i + +) do // Cost = c,, # times = (logn) + 1 
3. | q=qti? // Cost = c;, # times = log n 
4: end 

5: for (j = 1;j < n;j *= 2) do // Cost = c,, # times = (logn) + 1 
6 | q=q4+} // Cost = c3, # times = log n 
7: end 

s: for (k =n;k > 1;k/ = 2) do // Cost = c,, # times = (logn) + 1 
9 | q=q+k? // Cost = c}, # times = log n 
10: end 

Solution 


Note that this algorithm has three independent loops, each performing slightly distinct 
tasks. We have added side comments in gray showing the number of time each statement 
is executed. 
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The running time of the fragment is the sum of the cost multiplied by the number of times 
for each line. Therefore, 
Running time = c} +c,(1 + logn) + c, logn + 2c,(1 + logn) + 2c, logn 
(c1 +c, + 24) + (c, + 3c, + 2c,) logn = clogn + ĉ, 


where ¢ = c, + 3c, + 2c, and @=c, + c, + 2c,. Using the predominate term, the running 
time is clog n. a 


7.2.2 Types of Runtime Analysis 


A question that suggests itself is how do we evaluate the running time given there is con- 
ditional execution (i.e., selection statement)? Here, we will look at the worst-case running 
time. Why not the best-case? What would be the worst-case scenario be? There are three 
types of runtime analysis for a given algorithm: 


e Worst-case analysis: This type provides an upper bound on running time guaranteeing 
that the algorithm would not run longer. 

e Average-case analysis: This type provides a prediction about running time, assuming 
input is random. 

e Best-case analysis: This type provides a lower bound on running time on input for which 
algorithm runs the fastest. 


Note that the average running time can sometimes provide a more practical estimate 
of performance in real-world scenarios. However, calculating it is often significantly more 
challenging than determining the worst-case running time. 


Example 7.12 The code in Algorithm 7.18 sets small to the index of the smallest element 
found in the portion of the array A from A[i] through A[n — 1]. Find the running time 
equipped with worst-case performance. 


Solution 
We are looking at the worst-case running time. We have added side comments in gray show- 
ing the number of times each statement is executed. 

We consider the body of the “for” statement, the “if” statement (lines (3) and (5)). The 
test of line (3) is always executed, but the assignment at line (4) is executed only if the test 
succeeds. Thus, the body (lines (3) and (4)) takes either c} + c, or c, time costs, depending 
on the data in array A. If we want to take the worst-case, then we can assume that the body 
takes c, + c, time cost. 

The (worst-case) total cost or running time of the code is the sum of the cost multiplied 
by the number of times for each line. Therefore 


Running time = c +c,(n—i)+¢;(n-—i-—1)+c¢,(n—i-1) 
(cy +c, +€3)(n — i) — C3. 


It is natural to regard the size “n” of the data on which the code operates as n = n — i, 
since that is the length of the array A[i : n—1] on which it operates. Then the run- 
ning time, which is (c, + c, + c3)(n — i) — c}, equals cn — c} where ¢ = c} + c, + c}. To sum 
up, using the predominate term, the running time is cn where n = n —iandc = c) +c, + C3. 

a 
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Algorithm 7.18: The algorithm of Example 7.12 


1: sum =i // Cost = cy, # times = 1 
2: for (j =i+1;j<n;j++)do // Cost = c,, # times = n — i 
3: | if (A[i] < A[small]) then // Cost = c}, # times =n—i-1 
4: | sum =j // Cost = c,, # times = n — i — 1 [worst-case] 
5 | end 

6: end 


Algorithm 7.19: The algorithm of Example 7.13 


1: for (i = 1;i < m;i + +) do // Cost = c4, # times = m 
2: | if (i< n/2) then // Cost = c,, # times = m — 1 
3: while (n > 1) do // Cost = c,, # times = (m — 1)n [worst case] 
4 printf(n) // Cost = c,, # times = (m — 1)(n — 1) [worst case] 
5: n-— // Cost = cz, # times = (m — 1)(n — 1) [worst case] 
6: end 

7 end 

s: | while (n > 1) do // Cost = c,, # times = (m — 1)((log,) n) + 1) 
9 print£(n) // Cost = c,, # times = (m — 1) logo” 
10: n/ =10 // Cost = cg, # times = (m — 1) logo n 
u: | end 

12: end 


Example 7.13 Compute the worst-case and best-case runtime complexities of the pro- 
gram snippet shown in Algorithm 7.19. 


Solution 
First, we find the worst-case running time. The most-right column adds the number of 
times each statement is executed. 
The worst-case running time of the fragment is the sum of the cost multiplied by the 
number of times for each line. Therefore, 
Run time = cym+c,(m—1)+c,(m—1)n+(c,+¢;)(m— 1)(n — 1) 
+c,(m — 1)(1 + log, yn) + c,(m — 1)log,)n + cg(m — 1)log, yn 
= cM + (c +¢,)(m — 1) + c,(m — 1)n + (c4 +.¢5)(m — 1)(n — 1) 
+(¢5 + C} + ¢g)(m — log, gn. 
Using the predominate term, the running time is cam where € = c} + C4 + Cs. 
The best-case running time of the fragment is the sum of the cost multiplied by the 
number of times for each line. Therefore, 
Running time = cm + c,(m — 1) + cẹ(m — 1)(1 + logion) + c7(m — 1)log, yn 
+c,(m — 1)log, n 
= Cm + (C, + Cg)(M — 1) + (Ce + C7 + Cg)(m — 1)log, gn. 
Using the predominate term, the running time is êm log n where ¢ = (c + cz + Cg)/ log 10. 
a 
Another example for computing different case complexities is shown in Insertion Sort 
which will be introduced and analyzed in Chapter 8. 
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7.2.3 Summation Representations for Looping 


In all the illustrative instances, we have examined thus far concerning the assessment of 
running times, our approach has consistently involved a detailed examination of the code, 
performing a line-by-line analysis to determine the time complexities involved. In this 
part, we depart from the conventional practice of inspecting each line’s execution time 
individually. Instead, we pivot toward a more abstract perspective, providing additional 
illustrative examples that showcase the running times of loop-based programs. In this new 
approach, we represent the running time in the form of summations, allowing for a more 
generalized and efficient method of analysis. 

Note that the inner loop in Algorithm 7.15 of Example 7.9 is the part that contributed most 
to the running time. In fact, there is an informal rule called the 90-10 complexity rule which 
states that 90% of the running time is spent in 10% of the code. However, the exact percent- 
age varies from one program to another. Instead of analyzing every statement, we can save 
time and effort by focusing on the statement that contributes most to the running time. 


Example 7.14 Consider the code in Algorithm 7.20. It does not really matter what the 
code does, but what really matters is finding the running time of its execution. Instead of 
analyzing every statement, we can save time and effort by focusing on the statement in 
line (4). 

Assuming n is a constant, say 3, how many times would the statement in line (4) be 
executed? It is clear that the answer here is 3 x 3 = 3” = 9 times. Generalizing this for any 
n, we find that the statement in line (4) would be executed n? times. 

Let c be the execution cost of the statement in line (4). Then we can represent the running 
time using summations and obtain Xi ÈX- ¢ = ¢ Din n = en’. 

Thus, the running time of the code in Algorithm 7.20 is cn’. a 


Algorithm 7.20: The algorithm of Example 7.14 
1::x=0 
2: for (i = 1; i < n; i + +) do 
3: | for(j=1;j<n; j++)do 
4 | x=x+(i-j); // Execution cost is c 
5: | end 
6: end 
7: return x 


When endeavoring to determine the running time of an algorithm, there are some note- 
worthy considerations to bear in mind: 


e One can analyze algorithms by focusing on the part(s) where most of the execution 
occurs. 

e The running time is significantly influenced by the input data provided to the algorithm. 
Different inputs can lead to variations in the running time of the algorithm. 


Example 7.15 Find the running time of the code in Algorithm 7.21 without doing a line- 
by-line analysis. 
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Algorithm 7.21: The algorithm of Example 7.15 
1:x=0 
2: for (i = 1; i<n;i++)do 
3: | for(j=1;j<i;j++)do 
4: | x=x+(i-j); // Execution cost is c 
5: | end 
6: end 
7: return x 


Solution 

To save time and effort, we focus on the statement in line (4). Assume that n is a constant, 
say 3. We determine the number of times the statement in line (4) is executed by tracing the 
code and find that 


So, the number of times the statement in line (4) is executed is 1 + 2 + 3 = 6 times. 
Now, generally speaking, for any n, we determine the number of times the statement in 
line (4) is executed by tracing the code and find that 


Therefore, the number of times the statement in line (4) is executed is 


k n(n + 1) 
1+2+---+n= pis —— times, 
r 2 
i=1 
where we used the arithmetic series to obtain the last equality. 


Let c be the execution cost of the statement in line (4). Then, the running time of the code 
in Algorithm 7.22 is cn(n + 1)/2 x cn?. m 
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Example 7.16 Find the running time of the code in Algorithm 7.22 without doing a line- 
by-line analysis. 


Algorithm 7.22: The algorithm of Example 7.16 
1:x=0 
2: for (i = 1; i<n;i++)do 
3: | for (j =i; j <n; j++)do 
4 | x=x+(i-j); // Execution cost is c 
5 | end 
6: end 
7: return x 


Solution 

To save time and effort, we focus on the statement in line (4). Assume that n is a constant, 
say 3. We determine the number of times the statement in line (4) is executed by tracing the 
code and find that 


i j = i ton 
I 1 to3 = 3 
2 2 to3=2 


3 3 to3 = 1 


So, the number of times the statement in line (4) is executed is 3 +2 + 1 = 6 times. 
In general, for any n, we have 


i j = i ton 

1 1 ton=n 

2 2 to n = n-1 

3 3 to n = n-2 
n-1 n-1 ton = 2 

n n ton= 1 


Therefore, the number of times the statement in line (4) is executed is 


nea prm- ank a 


i=1 


times. 


Let c be the execution cost of the statement in line (4). Then, the running time of the code 


in Algorithm 7.21 is cn(n + 1)/2 x cn’. m 


7.2 Running Time of Algorithms 


Other examples of (double and triple) summation representations for looping are the 
matrix-vector multiplication and the matrix—matrix multiplication (see Section 8.1). 


7.2.4 Upper and Lower Bounds for Running Time 


Some summations are difficult to work with. In this part, we find upper and lower bounds 
for running time. This allows us to simplify the summations. 

In adopting this approach, we require that the upper and lower bounds must be the same 
function, only differing by a constant. The constant for the upper bound must be larger than 
or equal to the constant for the lower bound. 

To obtain an upper bound in this approach, we remove terms of expression being sub- 
tracted, if helpful, and substitute terms (usually, but not necessarily, the upper/top bound 
of the summation) into expression. To obtain a lower bound, we split summations to reduce 
size, if helpful, and substitute terms (usually, but not necessarily, the lower/bottom bound 
of the summation) into expression. 


Example 7.17 (Example 7.15 revisited) 

We find the upper and lower bounds for the running time of the code in Algorithm 7.21. 
Let c be the execution cost of the statement in line (4). Then we can represent the running 
time using summations and obtain 


n 


py = Yeon —it+1). 
=1 


i=1 j=i i 


Finding an upper bound is straightforward: 


n n 
Fen —i+1)< Yen = cn’. 
i=1 i=1 


We now find a lower bound: 
ye > Fan -i> Fen =i), (7.1) 
i=1 j=i i=1 i=! 


Our goal is to make the right-hand summation in (7.1) at least as large as a quadratic 
function. Let us try to substitute > or n for i and see what happens. 
Substituting > for i in (7.1), we get 


Sen-D< Ye(n—F). 


but this does not work because it makes the lower bound larger instead of smaller (for lower 
bounds, we should not make the lower bound larger). 
Substituting n for i in (7.1), we get 


but this does not work either because it makes the lower bound very small, it is zero. 
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Splitting the summation differently and using the lower split, we get 


Yin > Yan -9> Yre(n =) =e(2) (2) = (E)n 


Thus, the running time is ĉn? where c/4 < € < c. E 


Example 7.18 The running time of an algorithm is represented by the summation 


x: (7.2) 


where n is the number of times the outer loop is iterated and c is the execution cost of the 
statement that contributed most to the running time of the algorithm. Find upper and lower 
bounds for the running time of the algorithm. Can your upper/lower bound(s) be sharpened 
to obtain a tight bound for the running time? 


Solution 
Finding an upper bound is straightforward: 


n r n n 
E?e = c)n —i+1)< c)n =c. 
i=l jai i=l i=l 


For a lower bound, we need to make the summation representation in (7.2) at least as 
large as a cubic function. This is immediately obtained by noting that 


rye > So -i)> cS? -2> Fo -n)= =n —n’)> (£) n?. (7.3) 
i=1 i=" i=” 


i=1 j=i 
Although a lower bound was successfully obtained, the splitting was not helpful in (7.3). 
In fact, the above lower bound can be sharpened as follows. 


n r 


E?e > DI -i)> Soi —n) = c(n? —n’) > enè. (7.4) 
i= i=l 


i=1 j=i 
Therefore, a tight bound for the running time of the algorithm represented by the 
summation 


is cn>. This gives us the desired bound. a 


Note that, in obtaining lower bounds in Example 7.18 (where second inequality in (7.3) 
and the first inequality in (7.4) follow), we substituted n for i and did not get 0. This is the 
difference between Examples 7.17 and 7.18. 
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Example 7.19 Find upper and lower bounds for the running time of the algorithm 
represented by the summation 


wy: (7.5) 


i=1 j=l k=j 

where n is the number of times the outer loop iterated and c is the execution cost of the 
statement that contributed most to the running time of the algorithm. 

Solution 

For an upper bound, we have 


n oP P n 


EDD TEED e soi seyri = cn’. 


i=1 j=l k=j i=1 j=1 i=1 j=l 


For a lower bound, we need to make the summation representation in (7.5) at least as 
large as a quintic function. This is immediately obtained by noting that 


PIED IE 


i=1 j=1 k=j i=1 j=1 


> ey Ve -D 


IV 
fs 
M:= 
Ms 
Z 
N 
| 
| 
WV 


IV 
a 
l 


IV 
~ i) 
i Ms ] 
= 
3 
~s 
N 
— 
P 


n nt/16 c 
>c(|= = ( — ) n’. 
> e( 2 ) ( 4 ) ( 128 )n 
Therefore, the running time of the algorithm represented by the summation in (7.5) is cn’, 
where g ESO E 
7.3 Asymptotic Notation 


The asymptotic notation allows us to express the behavior of a function as the input 
approach infinity. In other words, it is connected about what happens to a function f(n) 
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as n gets larger, and is not concerned about the value of f(n) for small values of n. In this 
section, we introduce the notations, present their properties and criteria of selection, char- 
acterize the definitions of the notations using limits, and give a complexity classification 
of algorithms based on these notations. 


7.3.1 The Notations 


We present the definitions of Big-Oh, Big-Omega, and Big-Theta which are all asymptotic 
notations to describe the running time of algorithms. Throughout this section, unless it 
is stated explicitly otherwise, we assume that f and g are two asymptotically nonnegative 
functions on R, that is, f(n) and g(n) are nonnegative whenever n is sufficiently large. 


Definition 7.2 (Big-O) We say that f(n) is Big-O of g(n), written as f(n) = O(g(n)), if 
there are positive constants c and ng such that f(n) < cg(n) for all n > no. 


According to Definition 7.2, f(n) = O(g(n)) means that f(n) grows no faster than g(n). See 
Figure 7.4. 


Remark 7.1 The “=” in the statement “f(n) = O(g(n))” should be read and thought of as 
“is,” not “equals.” The reader can think of it as a one-way equals. An alternative notation 
is to write f(n) € O(g(n)) instead of f(n) = O(g(n)). 


Bearing in mind Remark 7.1, Definition 7.2 can be rewritten as follows. 


Definition 7.3 (Big-O revisited) Big-Oh of g(n), written as O(g(n)), is the set of func- 
tions with smaller or same order of growth as g(n). More specifically, 


O(g(n)) = {f(n) : J positive constants c andny s.t. 0 < f(n) < cg(n) Yn > no}. 


A < cgin) 
S 


fin) 


Figure 7.4 f(n) = O(g(n)). 


7.3 Asymptotic Notation 


Here g(n) is called an asymptotic upper bound for f(n). 
The following example illustrates Definitions 7.2 and 7.3. 


Example 7.20 Prove the following asymptotic statements. 


(a) n?+n= O(n’). (c) ni +n3logn = O(n?) 

(b) 3n3 — 2n? + 7n — 9 = O(n?). 

Solution 

(a) Ifn > 1, we haven < n?. It follows that n? + n < n? + n? = 2n? for all n > 1. Therefore, 
according to Definition 7.2, we have n? + n = O(n”) with Ny = landc=2. 

(b) Ifn > 1, we have that 3n? — 2n? + 7n — 9 < 3n? + 7n < 3n? + 7n? = 10n? forall n> 1. 
Thus 3n? — 2n? + 7n — 9 = O(n?) with ny = 1 andc = 10. 

(c) Ifn > 1, we have ni < nê and logn < n, which implies that 


ni +n? logn < nî + (n?)(n) = 2n'. 


Thus, with n, = 1 and c = 2, we have n: +n? logn = O(n’). a 


Below are some relationships among the growth rates of some common functions: 
c< logn < logn « Wn «n «Knlogn <n! «Kn? «K2" «3" «Kn! Kn". 


For instance, n? « 2” means that n? is asymptotically much less than 2”. In fact, when 
n = 10, we have 2" = 1024and 102 = 1000. Now, each time we add 1 to n, 2” doubles, while 
n? is multiplied by the quantity (+)3 which is less than 2 when n > 10. 


n 


Example 7.21 Prove the following asymptotic statements. 


(a) 2# +n? = O(2"). (c) (v/2)!8" 2 O(y/n). 
(b) log(n!) = O(n log n). 


Solution 
(a) Note that for n > 10, we have n°? < 2”. It follows that, for n > 10, we have 2” +n? < 
2” + 2” = 2(2”). Thus, 2” + n? = O(2") with n, = 10 andc = 2. 
n times 
(b) If n>1, we have n!=n(n—-1)---2-1<n-n---n-n=n", and hence log(n!) < 
logn” = nlogn. Thus, log(n!) = O(n logn) with ny) = 1 and c = 1. 
(c) Note that 


logn logn 1 
(AJE = (ah) aben at anh a 


Then it is clear that (v2) a O(y/n). a 


1 Computer scientists generally think of “log n” as meaning log,n rather than log, )n and log,n. 
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Definition 7.4 (Big-Q) Big-Omega of g(n), written as Q(g(n)), is the set of functions 
with larger or same order of growth as g(n). More specifically, 
Q(g(n)) Ê {f(n) : J positive constantsc andn, such that 
0 < cg(n) <f(n) Vn > no}. 


Here g(n) is called an asymptotic lower bound for f(n). 
Figure 7.5 illustrates Definition 7.4. We also present Example 7.22. 


Example 7.22 Prove the following asymptotic statements. 
(a) n? + 5n? = Q(n’). (b) nlogn -n +log9 = Q(n logn). 


Solution 

(a) Ifn > 1, we have n? < n? < n? + 5n?. Thus, according to Definition 7.4, we conclude 
that n? + 5n? = Q(n?) with ny = land c = 1. 

(b) We need to show that there exist positive constants c and n, such that cnlogn < 
nlogn -n + log9 for all n > ng. Since nlogn — n < nlogn—n+log9, we will instead 
show that 


cnlogn<nlogn—n, or equivalently c< 1- 


logn` 

1 1 = i 1 yee icki = 
If n > 4, then re < (as log4 = 2), or equivalently 5<1 Den? So, picking c = 5 
suffices. In other words, we have just shown that if n > 4, 


snlogn <nlogn—n. 
Thus, ifc = ; and ny = 4, then for all n > np, we have 
cnlogn < nlogn -n < nlogn -n + log9. 


Therefore, n logn — n + log9 = Q(n logn). E 


fín) 


cg(n) 


Figure 7.5 f(n) = Q(g(n)). 


7.3 Asymptotic Notation 


Now, we shift our focus to the Big-Theta notation. Early works by Donald Knuth 
and Robert Floyd in the 1960s and 1970s (see Knuth (1997), Floyd (1993)) significantly 
contributed to the formalization and widespread use of Big-Theta notation in the analysis 
of algorithms. Like Big-Oh notation, the Big-Theta notation is also a fundamental tool 
in the field of computer science for describing the asymptotic behavior and efficiency of 
algorithms. 


Definition 7.5 (Big-O) Big-Theta of g(n), written as O(g(n)), is the set of functions with 
the same order of growth as g(n). More specifically, 


O(g(n)) = {f(n) : I positive constantsc,,c, andn, such that 
0 <c,g(n) < f(n) < egin) Yn > no}. 


Here g(n) is called an asymptotically tight bound for f(n). 
Figure 7.6 illustrates Definition 7.5. We also present the following example. 


Example 7.23 Prove that n? + 3n + 6 = O(n°?). 


Solution 
When n>1, we have n? <n? +3n+6 <n? + 3n? + 6n? < 10n?. Thus, according to 


Definition 7.5, we conclude that n? + 3n + 6 = O(n?) with n, = 1 andc, = 1 and c, = 10. 
7 


What if some function is not Big-Oh, not Big-Omega, or not Big-Theta of some other 
function? The method of proving this is to assume that witnesses n, and c exist, and derive 
a contradiction. We have the following example. 


A 
e2g(n) 


fa) 


ci g(n) 


Figure 7.6 f(n) = @(g(n)). 
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Example 7.24 Show that n? is not O(n). 


Solution 

Suppose that n? = O(n). Then there exists ng and c such that n? < cn for all n > no. Picking 
n, = max {No, 2c}, we have n? < cn}, and dividing both sides by n}, we get n, < c. But we 
chose n; to be at least 2c, a contradiction. This concludes that n? is not O(n). E 


7.3.2 Properties of the Notations 


There are a lot of properties for Big-Oh, Big-Omega, and Big-Theta notations. In this part, 
we present a few of the most important ones. 

The following property follows almost immediately from Definitions 7.3, 7.4, and 7.5, and 
therefore its proof is left as an exercise for the reader. 


Property 7.1 f(n) = O(g(n)) iff f(n) = O(g(n)) and f(n) = Q(g(n)). 
Figure 7.7 shows a conceptual relationship among the notations. 


Example 7.25 Prove the following asymptotic statements. Do not prove by a direct use 
of Big-Theta definition (Definition 7.5). 


(a) in? + 4n = O(n’). 
(b) nÈ + 5n” — 13n? — 6n* + 4n? — 21 = O(n). 


Solution 

(a) Ifn > 1, we have in? +4n < in? + 4n? = 4.25n?. Therefore, in? + 4n? = O(n?). On the 
other hand, when n > 1, we have in? < in? + 4n. Therefore, in? + 4n? = Q(n?). Con- 
sequently, by Property 7.1, we conclude that in? + 4n = O(n?). 

(b) Let f(n) = n? + 5n” — 13n> — 6n* + 4n? — 21. To prove that f(n) = @(n5), we show that 
f(n) = Oln’) and f(n) = Q(n?). 
It is clear that when n > 1, we have 


f(n) < n? + 5n? +4n? < nÈ + 5n? + 4n? = 10n8. 


Thus, f(n) = O(n) as desired. 
Next, we prove that f(n) = Q(n®). That is, we need to show that there exist positive 
constants ng and c such that en? < f(n) for all n > ny. 


Figure 7.7 Relationships among Big-Oh, Big-Omega, and 
Big-Theta notations. 
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Now, because 


f(n) = n? + 5n’ —13n? — 6n* + 4n? — 21 
> në — 13n? — 6n’ — 21 
> n — 13n” — 6n’ — 21n” = nè — 4077, 


we will instead show that cnë < në — 40n” for some c > 0 and for all n > ny. 
Note that 

cn’ < n -40n > (1- on? > 40n” > c<1- =. 
So, ifn > 80, thenc = ; suffices. 
Thus, we have shown that if n > 80, then f(n) > inë, Thus, f(n) = Q(n?). This com- 
pletes the proof. a 


The following property follows almost immediately from Definition 7.3, and therefore its 
proof is also left as an exercise for the reader. 


Property 7.2 (Transitivity) If we have f(n) = O(g(n)) and g(n) =O(h(n)), then 
f(n) = O(h(n)). Same for Big-Omega and Big-Theta. 


Example 7.26 Itis easy to see that 4n? + 3n + 17 = O(n?) and that n? = O(n). Based on 
Property 7.2, we conclude that 4n? + 3n + 17 = O(n’). E 


Property 7.3 (Scaling by a constant) If f(n) = O(g(n)), then kf(n) = O(g(n)) for any 
k > 0. Same for Big-Omega and Big-Theta. 


Proof: Assume that f(n) = O(g(n)), then from Definition 7.3, there are positive constants c 
and n, such that f(n) < cg(n) for n > nọ. Multiplying with k > 0, we get kf(n) < cg(n), for 
n > No, where c = kc. The result is established for Big-O. Similar arguments can be made 
for Big-Omega and Big-Theta. E 


Example 7.27 Leta and b be positive numbers that are different from 1. Bearing in mind 
that log,n = log,n/log,a. Then, according to Property 7.3, it follows that logn = O(log,n). 
a 


Property 7.4 (Sums) If we have f,(n) = O(g,(n)) and f,(n) = O(g,(n)), then fi(n) + 
Jan) = O(g,(n) + g5(n)) = O(max {g,(n), g,(n)}). Same for Big-Omega and Big-Theta. 


Proof: We prove the result for Big-O, and similar arguments can be made for Big-Omega 
and Big-Theta. 

Let fi(n) = O(g,(n)) for i = 1, 2, then from Definition 7.3, there are positive constants c; 
and n; such that f,(n) < ¢,g,(n) for n > nj. 

Now, let nọ = max {n,,n,}, then for all n > ny, we have 


~- 


F) +f) < cg (n) + cgn) 
< c, max {g,(n),g,(n)} + c max {8 (n), g,(n)} 
= (cı + ¢,)max {g; (n), 8(n)} 
< cmax {g,(n), 8(n)} < egn) + 8,(n)), 
where c = 2 max {c,,c,}. The proof is complete. a 


We leave the proof of the following property as an exercise for the reader. 
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Property 7.5 (Products) If we have f,(n) = O(g,(n)) and f,(n) = O(g,(n)), then 
fmf.) = O(g,(n)g,(n)). Same for Big-Omega and Big-Theta. 
Example 7.28 In Example 7.20, we showed that 
n2+n=O(n2) and n? +n? logn = O(n’). 
Applying Property 7.4, we conclude that 
n? +nlogn+n2+n=0O (nt +n?) = O(n*). 
Applying Property 7.5, we conclude that 


(rÈ +m logn) on? +m) = O (x°) (x?) = 009) 


We also leave the proofs of Properties 7.6 and 7.7 as exercises for the reader. 


Property 7.6 (Symmetry) We have f(n) = O(g(n)) iff g(n) = O(f(n)). We also have 
f(n) = O(g(n)) iff gin) = QF (n)). 


Property 7.7 (Reflexivity) f(n) = O(f(n)). Same for Big-Omega and Big-Theta. 
Property 7.8 Iff(n) = O(g(n)) and g(n) = O(f(n)), then f(n) = O(g(n)). 
Proof: Let f(n) = O(g(n)), then there are positive constants c} and n, such that 
fmn) < egin), Vn > n. (7.6) 
Let also f(n) = O(g(n)), then there are positive constants c, and n, such that 
g(n) < o f(n), Yn > n,. (7.7) 
Combining (7.6) and (7.7), we obtain 
1 
zE < f(n) < cgn), Vn > No, 
2 


where nọ = max {n,,n,}. The proof is complete. E 
The following remark follows immediately from Properties 7.2, 7.6 and 7.7. 


Remark 7.2 The asymptotic notation © defines an equivalence relation over the set of 
nonnegative functions on N. 


From Theorem 2.1, © defines a partition on these functions, with two functions being in 
the same partition (or the same equivalence class) if and only if they have the same growth 
rate. For instance, the functions n?, n? + logn and 3n? +n + 1 are all @(n?). So, they all 
belong to the same equivalence class. 


7.3 Asymptotic Notation 


7.3.3 The Notations in Terms of Limits 


In this part, we characterize Big-Oh, Big-Omega, and Big-Theta notations in terms of limits. 
The limits are used as another technique to prove asymptotic statements, which is often 
much easier than that of definitions. It is not hard to prove the following theorem. 


Theorem 7.1 (Limit characterization of notations) Let f(n) and g(n) be two 


n 
asymptotically nonnegative functions on R such that tim, oE = L. Then 


(i) If L = 0, then f(n) = O(g(n)). (iii) If0 < L < œ, then f(n) = O(g(n)). 
(ii) If L = œ, then f(n) = Q(g(n)). 


If the limit in Theorem 7.1 does not exist, we need to resort using definitions or some 
other techniques. 


Example 7.29 Use limits to prove the following asymptotic statements. 


(a) n? = O(n’). (e) nD = O(n?). 

(b) n? = Q(n). (f) logn = O(n). 

(c) 7n? = @(n?). (g) n? = O(2"). 

(d) nt — 14n? + 8n? + 27n —9 = O(n’). (h) VIn? = 3n +2 = O(n). 
Solution 


We simply apply Theorem 7.1. 


(a) lim, soo = lim} =0. By Theorem 7.1(i), we have n? = O(n’). 
n? n>0n 
2 
(b) lim, so = limn = oo. By Theorem 7.1(ii), we have n? = Q(n). 
h noo 
2 
(c) LM yo = lim7 = 7 > 0. By Theorem 7.1(iii), we have 7n? = @(n?). 
n= 
(d) We have 
4 qAn3 2 = 
lim 2 14n° + 8n* + 27n > = lim (1 4,8 27 2) 
n>% n4 n>% n e B m 


=1-0+0+0-0=1. 


Thus, by Theorem 7.1(iii), we have nt — 14n? + 8n? + 27n — 9 = O(n’). 
(e) We have 
1)/3 
lim tt a 


n>% n? n>% 3n? n>% n2 n>œ 18n 


where we used L’Hospital’s rule twice to find the limit. Therefore, by Theorem 7.1(i), 


we have — = O(n?). 
(£) We have 
logn 1/n 
mo a ee. 
n>% Nn n=% n>o Nl 


where we used L’Hospital’s rule to find the limit. Therefore, by Theorem 7.1(i), we have 
logn = O(n). 


207 


208 


7 Analysis of Algorithms 


(g) We have 


_ Ww . 2 . . 
lim — = lim —— = lim ———~ = lim ———— = 
n>%02” n>%2"]ln2 n>o2"(In2)2 n>02"(1n2)3 


> 


where we used the L’Hospital’s rule three times to find the limit. Therefore, by 
Theorem 7.1(i), we have n? = O(2"). 


(h) We have 
V7n2 — 3n +2 2 
a = jim = 3042 Sn+2_ fim (7-442) =i 
n>% n n>% n n>% n n 
Thus, by Theorem 7.1(iii), we have y7n? — 3n + 2 = O(n). 
The desired proofs are obtained. a 


7.3.4 Complexity Classification of Algorithms 


In this part, we give the computational complexity classification of algorithms based on the 
notations. First, we discuss the properties that we should consider to choose the Big-Oh of 
algorithms. 


7.3.4.1 Choosing Big-Oh for Algorithms 
For any algorithm, our Big-Oh choice will have two major properties: simplicity and 
tightness. 


e Simplicity: Our choice of a Big-Oh bound is the simplicity in the expression of the func- 


tion. We have the following definition. 


Definition 7.6 Let g(n) be a Big-Oh bound on f(n), that is, f(n) = O(g(n)). The function 
g(n) is said to be simple if the following two conditions hold: 


(i) It is a single term. (ii) The coefficient of that term is one. 
The following example illustrates Definition 7.6. 
Example 7.30 In Example 7.9, we showed that the running time of the code in 
Algorithm 7.15 is 

f(n) = tn? + en +4, 


where € = c, +3,¢ = 2c,,¢é=c, +c, and ç is the execution cost of line (i) in Algo- 
rithm 7.15 fori=1,...,4. 
Define 


g,(n) =n’, g,(n) =cn?, and g,(n) = Cn* + ĉn. 


Then, we can show that f(n) = O(g,(n)) for each i = 1,2, 3. Note that g} is simple, but g, 
and g, are not simple functions. So, choosing g,(n) as a Big-Oh bound on f(n) is better 
than choosing g,(n) and g,(n). a 


7.3 Asymptotic Notation 


e Tightness: We generally want the “tightest” Big-Oh upper bound we can prove. We have 
the following definition. 


Definition 7.7 Let g(n) be a Big-Oh bound on f(n), that is, f(n) = O(g(n)). The function 
g(n) is said to be a tight bound on f(n) if the following condition holds: 


(Ahm), fn) = O(h(n))) —> (g(n) = O(A(n))). (7.8) 


The implication in (7.8) means that if we can find a function h(n) that satisfies the state- 
ment that f(n) = O(h(n)), then the statement g(n) = O(h(n)) is also satisfied. In other 
words, according to Definition 7.7, g(n) is a tight Big-Oh bound on f(n) if g(n) is a Big-Oh 
bound on f(n) and we cannot find a function that grows at least as fast as f (n) but grows 
slower than g(n). The following example illustrates Definition 7.7. 


Example 7.31 (Example 7.30 revisited) 

In Example 7.30, we found that the function n? is a simple Big-Oh bound on f(n) = cn? + 
én + č. In this example, we use Definition 7.7 to show that n? is a tight bound on f(n), 
while n? is not a tight bound on f(n). 

To show that n? is a tight bound on f(n), suppose that there exists a function h(n) such 
that f(n) = O(h(n)). Then there are positive constants c and ng such that f(n) = cn? + 
ên + Č < ch(n) for all n > no. Then h(n) > Cn? for all n > no, which in turn implies that 
n? > (<)h(n) for all n > ny. This means that n? = O(h(n)). According to Definition 7.7, we 
conclude that n? is a tight Big-Oh bound on f(n). 

To see why the function n? is not bound on f(n), we pick h(n) = n?. We have see that 
f(n) = O(A(n)), but it is clear that n? is not O(h(n)). E 


One can prove the following theorem. 
Theorem 7.2 Iff(n) = O(g(n)), then g(n) is a tight Big-Oh bound on f(n). 


Example 7.32 In Example 7.25, we proved that in? + 4n = @(n’). By Theorem 7.2, we 
conclude that n? is a tight Big-Oh bound on in? +4n. E 


Based on the asymptotic notations introduced in this section, we can now express the 
worst-, average-, and best-case time complexities (i.e., running times) of algorithms using 
Big-Oh, Big-Theta and Big-Omega notations. In Chapters 7-9, we will analyze different 
programs by expressing their running times using the asymptotic notations. 


7.3.4.2 Classification of Algorithms Based on the Notations 

Time complexity in algorithm analysis represents the overall time needed for a program 
to execute until its completion. Deriving an exact formula for the total runtime function, 
denoted as f(n), can be a complex or even impractical endeavor. Often, we can significantly 
simplify this task by employing Big-Oh notation. In essence, algorithm time complexities 
are typically denoted using the Big-Oh notation, expressed as O(g(n)), which serves as an 
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upper bound on f(n). These complexities are categorized based on the type of function 
found in the asymptotic notation. In this part, we classify the algorithms according to this. 


Definition 7.8 Algorithms with running time: 


(i) ©(n) are said to have linear complexity and they are called linear-time algorithms. 
(ii) O(n?) are said to have quadratic complexity and they are called quadratic-time 
algorithms. 
(iii) @(n*), for some constant k, are said to have polynomial complexity and they are called 
polynomial-time algorithms. 


Note that, as n doubles, the run time doubles in the linear-time algorithms and quadru- 
ples in the quadratics-time algorithms. In Example 7.13, we found that, when the best-case 
analysis is considered, the algorithm has linear-time, while when the worst-case analysis is 
considered, the algorithm has linearithmic-time. Note also that linear and quadratic-time 
algorithms are special cases of polynomial-time algorithms. The following definition 
identifies precisely the efficiency of an algorithm. 


Definition 7.9 An algorithm is called efficient if it runs in polynomial time. 


Therefore, when we assert the existence of an efficient algorithm to address a problem, 
we usually refer to a polynomial-time algorithm. In Table 7.1, we list some of the more 
common running times for programs and their names. 

The set of all decision problems that can be solved with worst-case polynomial time- 
complexity is said to be the complexity class P. There are other complexity classes to be 
introduced. One of the most important among them is so-called NP-complete class, which 
will be introduced in Section 7.7. 


Table 7.1 Some common time complexities. 


Running time Name 

o) Constant time 
Odog n) Logarithmic time 
O(log*n) Polylogarithmic time 
O(n) Linear time 

O(n log n) Linearithmic time 
O(n?) Quadratic time 
O(n?) Cubic time 

o(n*) Polynomial time 
O(2") Exponential time 
O(n!) Factorial time 


O(n") Super-exponential time 


74 Analyzing Decision-Making Statements 
7.4 Analyzing Decision-Making Statements 


In this section, we analyze and identify the time complexity of the decision-making state- 
ments given in Subsection 7.1.1. We consider only basic statements such as if-statement, 
for-statement, while-statement and others. 


7.4.1 Simple Statements 


The time complexity for a simple statement, which includes assignments, reads, writes, or 
jump statements, is O(1). 


7.4.2 \f-Statement 


Upon initial examination of Figure 7.8, we find that an upper limit on the running time of 
an if-statement can be expressed as O (1 + max {fi (n), AONE The “+1” factor corresponds 
to the testing process. Since both f; (n) and f,(n) are positive for all values of n, we can omit 
the “1+” and conclude that the running time of the if-statement is O (max fin), fa) , as 
presented in Figure 7.8. 


7.4.3 For-Statement 


Upon initial examination of Figure 7.9, we find that an upper limit on the running time 
of a for loop can be expressed as O(1 + (f(n) + 1)g(n)). The factor f(n) + 1 signifies the cost 
of going around once, encompassing the body, the test, and the reinitialization. The “1+” 
at the beginning accounts for the initial initialization and the possibility of the first test 
being negative, resulting in zero iterations of the loop. In the common scenario where both 
f(n) and g(n) are positive, the running time of the for-statement simplifies to O(g(n)f(n)), 
as outlined in Figure 7.9. 


7.4.4 While-Statement 


Upon initial examination of Figure 7.10, we find that an upper limit on the running time 
of a while loop can be expressed as O(1 + (f(n) + 1)g(n)). The term O(f(n) + 1) serves as an 


/ O(1) a O(max {f,(7), fo(n) }) 


else-part 


O(f\(1)) O(fa(n)) 


Figure 7.8 The running time complexity of an if-statement. 
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Figure 7.9 The running time 
complexity of a for-statement. 


O(1) 


O(sin) fm) 


O(f(n)) 


At most 
g(n)-times 


around 
reinitialize | O(1) 
~~] Figure 7.10 The running time 
| complexity of a while statement. 
O (sin) fn ) 
At most 


Of(n)) 


g(n)-times 


around 


upper bound on the running time of both the body and the test after the body. Additionally, 
the “1” at the beginning of the formula accounts for the test that occurs before entering the 
loop. In the typical scenario where both f(n) and g(n) are positive, the running time of the 
while statement simplifies to O(g(n)f(n)), as outlined in Figure 7.10. 


7.4.5 Do-While-Statement 


After a first look at Figure 7.11, we find that an upper limit on the running time of a do- 
while loop can be expressed as O((f(n) + 1)g(n)). The “+1” term signifies the time required 
to compute and test the condition at the end of each iteration of the loop. It is worth not- 
ing that in this context, g(n) is always positive. In the common case, where f(n) is positive 
for all values of n, the running time of the do-while statement simplifies to O(g(n)f(n)), as 
indicated in Figure 7.11. 


7.5 Analyzing Programs Without Function Calls 


Figure 7.11 The running time 
complexity of a do-while statement. 


O(f(n)) 


At most O(1) 
g(n)-times 


around 


ofgin \f(n ) 


7.4.6 Block 


A first look at Figure 7.12 indicates that an upper bound on the running time of a lock is 
(0) (f (n)+:+ fan). Using the Big-Oh rules, this can be simplified as the one stated in 
Figure 7.12. 

In summary, below are the major points we should take from above: 


e The bound for a simple statement is O(1). 

e The running time of a selection statement is the time required to make a decision regard- 
ing which branch to pursue, in addition to the greater of the running times of the available 
branches. 

e The running time of a loop is determined by calculating the time it takes to exe- 
cute the loop’s body, including any control steps like reinitializing the loop index 
and comparing it to the limit. This running time is then multiplied by an upper 
bound on the number of iterations the loop can undergo. Additionally, any actions 
performed just once, such as initialization or the initial termination test (if the loop 
may not iterate at all), are added to this total if the loop could potentially have zero 
iterations. 

e The running time of a sequence of statements can be calculated by summing the running 
times of each individual statement. Frequently, one of these statements will have a sig- 
nificantly greater running time compared to the others, and according to the summation 
rule, the overall running time of the sequence is essentially determined by the dominant 
statement. 


7.5 Analyzing Programs Without Function Calls 


In this section, we analyze the running time of programs that do not contain function calls 
(other than library functions such as “printf”), leaving the matter of analyzing programs 
with function calls to Section 7.6. 

We use a graph- or tree-based running time analysis to derive Big-Oh upper bounds on 
the running time of programs without function calls (also known as sequential programs). 
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Figure 7.12 The running time 
| complexity of a block. 


Of) ++ fan) ) or O (max fn) 


A graph structure, or a flowchart, is a graphical representation of the separate steps of a 
program in sequential order. A tree structure is a graph structure where loops are clustered 
as nodes instead of cycles. In this section, we create and use graph and tree structures to ana- 
lyze and identify the running time of sequential programs (i.e., programs without function 
calls). We have the following examples. 


Example 7.33 Draw the graph structure for the code in Algorithm 7.23. Give a Big-Oh 
upper bound on the running time of each compound statement in Algorithm 7.23, as a 
function of n. What is the total time it takes for the function to run? 


Algorithm 7.23: The algorithm of Example 7.33 
ri=n 

2: while (i < n?) do 

3: i=it+ yn 

4: end 

s: if (i < 3n) then 

6 | printf(“The exam is multiple choice.") 

7: end 

8: else 

9: | print£(“The exam is essay.") 


10: end 


Solution 

The graph structure is shown in Figure 7.13. The while loop of lines (2) and (3) in Algorithm 
7.23 may be executed as many as n3/2-times, but no more. To see this, note that the initial 
value of i is n, and that in each iteration we increment i by yn. We keep executing the while 
loop until i is no longer smaller than n?. Therefore, the number of times we go around the 
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OU) 


o(ayn’?)) = O(n”) o(ı E T 1)= O(n?) 


O (max {1,4 i)= O(1) 
exam is 
essay 
O(1) 


Figure 7.13 The graph structure with time complexity for the code in Algorithm 7.23. 


At most 
ni-times 
around 


O(1) 


while loop is k where k is the largest positive integer so that 


nt+Vnt+yVn+---+Vn=n+kVn <r’. 
Sc 
k-time 
Hence k < (n? — n)/ yn = O(n?/?). Now, according to the summation rule, we can deter- 
mine that the running time of Algorithm 7.23 is bounded by O(n*/*). This bound is estab- 
lished by considering the maximum running time among various parts of the algorithm, 
including the assignment in line (1), the while loop in lines (2) and (3), and the selection 
statement in lines (5) through (10). The running time found in this example is the worst-case 
performance of the code in Algorithm 7.23. a 


Example 7.34 The function in this example determines whether a given integer n is 
prime or not prime. A positive integer n is said to be prime if it is divisible by only 1 
and itself. For example, 7 is a prime number. So, if n is not a prime, then it is divisible 
evenly by some integer i between 2 and yn. The primality test function is shown in 
Algorithm 7.24. 

Draw the tree structure for this algorithm. What is the upper bound on the run- 
ning time, expressed using Big-Oh notation, for each compound statement within the 
algorithm, in terms of n? Additionally, what is the overall running time of the entire 
function? 
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Algorithm 7.24: Primality test function: prime(int n) 
Input: A positive integer n 
Output: A TRUE/FALSE answer to Question “Is n a prime number?" 


1: inti 

2 i=2 

3: while (i? < n) do 

4: | if (n%i == 0) then 
s: | | return FALSE 
6: | end 

7. | else 

8: | i++ 

9: | end 

10: end 


11: return TRUE 


Solution 

The illustration of the tree structure can be observed in Figure 7.14. As delineated within 
Figure 7.14, the time taken by the if-else statement amounts to O(1), thereby establishing the 
runtime of the while-statement as O( yn). Note that each of the remaining lines constitutes 
singular statements and consequently operates in O(1) time. This realization culminates in 
an understanding that the collective runtime performance of the entire function converges 
and is encapsulated by the notation O( yn). a 


O(n) 


ody 


(LyaJ-1) 
times if-else 


(4)-(7) 


around 
O(h- 1) 
or O(n) 


O(1) 


Figure 7.14 The tree structure for Algorithm 7.24 with running time complexity. 


7.5 Analyzing Programs Without Function Calls 


Example 7.35 Algorithm 7.25 is an example of the so-called interior-point algorithm and 
is used for solving linear (and nonlinear) programs.” 


(a) No matter how the algorithm works and how the mathematical notations therein look, 
the graph structure (flowchart) of the algorithm can be visualized. Draw the graph 
structure for Algorithm 7.25. 

(b) Let N,,, be the number of times we go around the inner while loop of line (8). In view 
of the graph structure of the algorithm drawn in item (a), determine a good asymptotic 
upper bound on the number of iterations needed to obtain a desired solution using 
Algorithm 7.25 in each of the following cases. 

i) Nin = O(nm) and y E (0, 1) is an arbitrarily chosen constant. 

ii) Nj, = OC) andy =1- o//nm, where o > 0. 
In each case, give your answer as a function of n, m, vy, e€ using Big-Oh, and justify 
your determination. 


Algorithm 7.25: Interior-point linear optimization algorithm 
Input: Linear program with an n x m matrix A and a barrier function f 
Output: An e-optimal solution to the linear program 

x initialize e > 0,y € (0,1), 0 > 0, 8 > 0, x®, pW > 0,4 

2: set x 2x y 2 pO, 4 £ 40 

3: while (u > e) do 

4: | compute g = Vf(u,x) —AlA 


compotas (7 = (vp) "AT (A (v3) TAT) A (n) (2) 


-1 
6 | compute Ad = (A (V2f(H,x))' AT) A (Vf x) @) 


,, | compute 5(u,x) = 4/ T (AT VF, Ax) 


s: | while (ô > f) do 


9: set x Ê x+60Ax 
10: setdA2A+0AA 
1: compute g = Vf(u, x)— ATA 
>. | | compute Ax =- (7 - (Vf)? A" (A (VA) AT) A (v27) (g) 
i: | | compute Aã = (A (Yf, 9) AT] A (Vf) E 
iā compute 6(y, x) = 1 (Ax V2F(H, x)(Ax) 
Vu 
1s: | end 


16: | sety yp 


17: end 


2 Linear programming (also called linear optimization) is a method to achieve the best outcome (such as 
maximum profit or lowest cost) in a mathematical model whose requirements are represented by linear 
relationships (see Chapter 10 for more detail). 
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Begin algorithm 


y 


initialize e, y, 0, p, x, uw, 4®, f, A 


y 


xê x®, u 4 pO, a = 4 


y 


Is u 2 e? 

yes ¥ 
compute g 
compute Ax 
compute Ad 


y 


compute ô(u, x) 


y 


Is ô > p? |Z 


yes 4 


x£ x+0Ax 


Nou-times 


A+=A+OAA 


j 


compute g 


j 


compute Ax 


y 


compute Ad 


y 


compute ô(u, x) compute u * yu 


Nin-times 


End algorithm 


Figure 7.15 The graph structure of Algorithm 7.25. 


Solution 

(a) The graph structure of Algorithm 7.25 is shown in Figure 7.15. 

(b) Let Noy (respectively, N,,,) be the number of times we go around the outer (respectively, 
inner) while loop, and N be the number of iterations needed to obtain a desired solution 
using Algorithm 7.25. Then, in view of the graph structure of the algorithm drawn in 
item (a), we have 


N= Nin Nout: 


We now estimate N,,,,- Let uv be the parameter at the i iteration. Then we have 


pO yu) yu? aire yp. 
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Thus pv will be less than the given e > 0 if yiu® < e. Note that 


(0) 


Yu <e => y <e/p <= ilogy = logy’) < log(e/y). 


Since y € (0, 1), we have 
., logte/u!) _ logue) 
logy -logy ` 
Hence, 
log(u /e) 
-logy ` 


out = 


(i) Itis given that Ni, = O(nm). Now, ify € (0, 1) is an arbitrarily chosen constant, then 
y = O(1), and hence 


pO 
Nour < log (5) O(1). 
€ 


Thus, the number of iterations needed to obtain a desired solution is 


u® 
N = Np Nout = O (nm log (<)) : 
(2 


(ii) It is given that Ni, = O(1). Now, ify = 1 — o / ynm (o > 0), then? 


logy =log(1-0/ynm) z -o/V/nm, 


and hence 


log(u/e) _ log(u/e) (= >) 
Noea = g = y/nm1 O(1). 
t —logy o/ er a m log ( ) 


Thus, the number of iterations needed to obtain a desired solution is 


pO 
N=N,,Nout = O (vam m log (2 J): 


In Example 7.35, having a good asymptotic bound on the number of iterations required 
to obtain a desired solution using Algorithm 7.25, we can find a good asymptotic time com- 
plexity of the algorithm if the running times of lines (1)-(17) in Algorithm 7.25 are provided. 

Other examples of sequential programs are Linear Search and Selection Sort which will 
be introduced and analyzed in Chapter 8. More specifically, a graph structure will be drawn 
to analyze Linear Search, and a tree structure will be drawn to analyze Selection Sort. 


7.6 Analyzing Programs with Function Calls 
In this section, we analyze the running time of programs or program fragments that contain 


function calls. The function call can be recursive or nonrecursive. We have the following 
definition. 


3 For small positive values of x, we have log(1 +x) x (1 +x) -1 =x. 


219 


220 


7 Analysis of Algorithms 


Definition 7.10 A function that calls itself recursively is called a recursive function. 
A function that does not call itself but calls other functions is called a nonrecursive 
function. 


We start with analyzing nonrecursive programs. 


7.6.1 Analyzing Nonrecursive Programs 


A nonrecursive program is the one that contains nonrecursive functions. The following 
example analyzes a nonrecursive program that contains a nonrecursive function. 


Example 7.36 Consider the fragment shown in Algorithm 7.26. Here f (n) isafunction 
call. Give a simple and tight Big-Oh upper bound on the running time of Algorithm 7.26, 
as a function of n, on the assumption that: 


(a) The running time of £ (n) is O(1), and the value of £ (n) is 0. 
(b) The running time of £ (n) is O(n?), and the value of £ (n) isn. 


Solution 

The first statement is an assignment statement with constant time. The inside of the for loop 
is also a simple statement with constant time. So, the part that will affect overall runtime is 
the runtime and value of f(n). 

In item (a), the value of f(n) is 0, so the loop will never execute the body. It will only 
evaluate the condition once. Since the runtime of f(n) (and thus the runtime of checking 
the condition) is O(1), the overall runtime is O(1). 

For item (b), Figure 7.9 shows how the runtime of for loops is calculated. We add the 
initialization runtime (usually O(1)) plus the cost of going around the loop once multiplied 
by the number of times we go around the loop, represented O(1 + (f(n) + 1)g(n)). We keep 
in mind that “the cost of going around the loop once” is represented in Figure 7.9 as the 
runtimes of “test” plus “body” plus “reinitialize,” with “test” being the condition of the for 
loop and “body” obviously being the body. Now, we can apply this to our problem. The“test” 
runtime in the for loop is O(n7), since that is how long it takes f(n) to run. The “body” of 
the for loop takes O(1), as it is just a simple statement. The “reinitialization” runtime still 
takes O(1). We perform the loop O(n) times, since the value of f(n) is n. Thus, putting it all 
together, we have O(1 + (n? + 1 + 1)n) = O(n’). a 


Algorithm 7.26: The algorithm of Example 7.36 
1: sum = 0 
2: for (i = 1;i < f(n); i + +) do 
3: | sum + =i 
4: end 


When determining the time complexity or running time of a program that comprises 
nonrecursive functions, a systematic process is employed. This process involves a sequen- 
tial set of steps designed to ascertain the individual running times of these functions. The 
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procedure given in the following workflow, followed by Example 7.37, will teach us how 
to determine the runtime of a program consisting of nonrecursive functions. 


Workflow 7.1 There are three steps involved in the process of determining the runtime 
of a program consisting of nonrecursive functions: 


(i) Calculate the running times of functions that do not call any other functions. 
(ii) Calculate the running times of functions that call only functions for which you have 
already determined their running times. 
(iii) Proceed in this manner until you have computed the running time for all functions. 


To further clarify the steps provided in Workflow 7.1, we present the following example, 
which is a restatement of Example 3.23 in Aho and Ullman (1994). 


Example 7.37 Calculate the time complexity of the program in Algorithm 7.27. 


Algorithm 7.27: The algorithm of Example 7.27 


1: # include <stdio. h > 
2: int cat(int x, int n) 
3: int cow(int x, int n) 
// This line is left intentionally blank 
4: main() 
5: inta,n 
6: scanf(“%d", &n) 
7: a=cow(0, n); 
printf(“éd n",cat(a, n)) 
// This line is left intentionally blank 
s: int cat(int x, int n) 
9: inti 
10: for (i = 1;i < n;i + +) do 
11: | x+ = Í; 
12: end 
13: return x 
// This line is left intentionally blank 
14: int cow(int x, int n) 
15: inti 
16: for (i = 1;i<n;i++)do 
17: | x+ = cat(i, n); 
18: end 
19: return x 
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Solution 
Note that this program is nonrecursive. Figure 7.16 provides main 
a basic overview of the program outlined in Algorithm 7.27. 


Considering Figure 7.16, we first analyze the function cat, 

which does not call any other function. Next, we analyze the cow 

function cow, which only calls the function cat. Finally, we 

analyze the function main, which calls both the functions y 

cow and cat. 
It is evident that the function cat has a time complexity 


of O(n). Now, if a function call is within the body of a for Figure 7.16 A basic scheme 
loop, we add its cost to the time complexity for each iteration. of the program shown in 
Consequently, the running time ofa call to cowis O(n”). Fur- Algorithm 7.27. 

thermore, when the function call is within a simple statement, we add its cost to the cost of 
that statement. Thus, the function main takes O(n”) time. Consequently, the overall time 
complexity of this program is O(n’). a 


7.6.2 Analyzing Recursive Programs 


A recursive program is the one that contains recursive functions. Analyzing the running 
time of a function that employs recursive calls demands more effort than evaluating non- 
recursive functions. Running time of a recursive program is represented by a recurrence. 
Remember that a recurrence is a mathematical equation or inequality used to express a 
function’s behavior based on its values for smaller inputs. 
For example, the following recurrence 


TQ) =1, Tin) = 3T(n-1)4+4, n=2,3,..., (7.9) 
describes the function (see Exercise 5.2 (b)) 
T(n) = 3"-2, n=1,2,.... (7.10) 


The following are some examples of recurrence formulas commonly found in recursive 
algorithms: 


e A recursive algorithm that iteratively processes the input by removing one item at each 
step. For example: 


T(n) = Tn-1)4+1, Tin) = T(n — 1) + n, etc. 

e Arecursive algorithm that halves the input size. For example: 
T(n) = T(n/2)+ 1, T(n) = T(n/2) +n, etc. 

e A recursive algorithm that divides the input into two equal halves. For example: 
T(n) = 2T(n/2) + 1, etc. 


The question that naturally arises now is: how do we determine the actual running time 
of a recursive program? The answer to this question is by following three steps: 


e First, derive a recurrence relation or formula from the given recursive program. This 
relation should express the time complexity of the problem size in terms of the time 
complexity of smaller instances of the same problem. 

e Next, solve the recurrence formula to find an explicit expression of the recurrence. 
Different methods for solving recurrences were discussed in Chapter 5. 


7.6 Analyzing Programs with Function Calls 


e Last, bound the recurrence explicit formula by an asymptotic expression that involves n. 
For instance, from (7.10), it is clear that the running time for the recursive program of the 
recurrence formula (7.9) is O(3”). 


In the following examples, we introduce two recursive programs, derive their recurrence 
formulas, and analyze their running times. 


Example 7.38 (Factorial) 

The recursive program in Algorithm 7.28 computes the factorial function n!. Assuming 
the cost times of the simple statements return 1 in line (3) and return n in line (6) 
are c} and c,, respectively, and observing that the function fact (n-1) that the program 
calls in line (6) is the same problem but with size (n — 1), we conclude that the cost time of 
executing line (6) is the constant time c, plus the time taken by the function fact(n — 1). 
Let T(n) be the running time of Algorithm 7.28 on an integer n. It follows that the recurrence 
formula for this recursive program is 

TQ) = c, 
T(n) =c,+T(n-1), n>1. 


From Example 5.3, we have 


(7.11) 


T(n) = (n — 1)c, + T(1) = (n — De, +c. 


Algorithm 7.28: Factorial function: fact(int n) 
Input: A positive integer n 
Output: The factorial n! 
1: fact(intn) 
2: if (n < 1) then 
| return 1 


w 


4: end 
: else 
6: | return n*fact(n — 1) 


wn 


7: end 


Thus, the recurrence formula (7.11) of Algorithm 7.28 describes the linear function 
T(n) = c&n + (c —¢,) for n>1. As a result, the factorial function algorithm is linear, 
running in O(n) time. ic] 


Example 7.39 (Integer Power) 

The recursive program in Algorithm 7.29 computes x”. We assume that the cost time of the 
simple statement return 1 in line (3) is c}, and that the cost time of each of the simple 
statements return p*p in line (8) and return x*p*p in line (11) is c,. Note that the 
function pow (x,n/2) that the program calls in line (7) is the same problem but with size 
n/2, and that the function pow (x, (n—1)/2) that the program calls in line (10) is the same 
problem but with size (n—1)/2. Let T(n) be the running time of Algorithm 7.29 on an integer 
n, then the recurrence formula for this recursive program is 


C1, ifn=1; 
T(n) = 4 T(n/2) + c, if n > 1 and is even; (7.12) 
T(n—1)/2)+¢,, ifn > 1 and is odd. 
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Since the function T is monotonically increasing, the recurrence formula (7.12) can be 
simplified to 
T(n) = T([n/2]) +c, if n>1, 
and hence 
T(n) < T(n/2) + c. 
The same method that was used in Example 5.4 can be used to show that 
T(n) < kc, + T(n/2*) <c,logn+T(1) = c logn + c. 


Thus, the recurrence formula (7.12) of Algorithm 7.29 describes the logarithmic function 
T(n) = cœ logn + c. As a result, the integer power algorithm is logarithmic, running in 
O(log n) time. E 


Algorithm 7.29: Integer power function: pow(int x,int n) 
Input: Positive integers x and n 
Output: The nth power of x 

1: pow(int x, int n) 

2: if (n == 0) then 


3: return 1 

4: end 

5: else 

6 | if (n%2 == 0) then // Checking if n is even 
7: int p = pow(x% n/2) 

8: return p * p 

9: else 

10: int p = pow(x, (n-1)/2) 
11: return x * p * p 

12: end 

13: | end 

14: end 


Other examples of recursive programs are Binary Search and Merge Sort which will be 
studied in Chapter 8. At the end of this section, we note that sometimes it is convenient to 
distinguish common recursive algorithms as in Table 7.2. 


7.7 The Complexity Class NP-Complete 


In 1971, Leonid Levin independently, along with Stephen Cook, made a significant 
discovery in the field of computational complexity theory. They introduced the concept 
of NP-complete problems by demonstrating that the satisfiability problem? falls into the 


4 The satisfiability problem was introduced in Section 1.5. 


7.7 The Complexity Class NP-Complete 


Table 7.2 Some common recurrence formulas and their complexities. 


Divide and conquer algorithms 


T(n) = aT(n/b) + f(n), (a> 1 and b > 1) 


T(n) = T(n/2)+ c 
T(n) = T(n/3)+ c 
T(n) = T(n/2)+ cn 
T(n) = T(n/3)+ cn 
T(n) = 2T(n/2) + cn 
T(n) = 3T(n/3) + cn 
T(n) = 3T(n/2) + cn 
T(n) = 4T(n/2) + cn 
T(n) = 2T(n/2) + cn? 
T(n) = 4T(n/2) + cn? 


Chip and conquer algorithms 
T(n) = T(n — a) + f(n), (a> 1) 


T(n)=T(n-1)+c 
T(n)=T(n-1)+cn 
T(n) = T(n — 1) + cn? 


Exponential algorithms 


T(n) = 2T(n — 1) + f(n) 
T(n) = 3T(n — 1) + f(n) 
T(n) = 4T(n — 1) + f(n) 
T(n) = 2T(n — 2) + f(n) 
T(n) = 2T(n — 3) + f(n) 
T(n)=T(n-1)+T(n-2)+f(n) 


T(n) = T(n-1)+T(n-2)+T(n-3)+f(n) 


T(n) = Yi TO+fm 


Running complexity 
time 


O(log n) 
O(log n) 
@Q(n) 

O(n) 

O(n log n) 
O(n log n) 
@(n'°s2@)) 
ond) 
on?) 
O(n? logn) 


Running complexity 
time 


O(n) 
O(n?) 
O(n?) 


Running complexity 
time 


Q(2") 
Q(3") 
Q(4") 
Q2") 
Q2”) 
Q2" 
Q2" 
Q2"/) 


category of NP-complete problems. NP-complete problems play a crucial role in compu- 
tational complexity theory, specifically in the domain of decision problems. Therefore, 
in this section, we redirect our attention from “algorithms” to “problems.” We have the 


following definitions. 


Definition 7.11 An abstract problem is a binary relation on a set of problem instances 


and a set of problem solutions. 
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For instance, consider the shortest path problem.° In this problem, we are given an 
undirected graph, denoted as G, along with two specific vertices, say u and v, within that 
graph. The objective is to determine a path between u and v that employs the fewest 
possible edges. The problem itself represents a binary relation. It associates each instance 
consisting of a graph and two vertices with a shortest path within that graph, connecting 
the specified pair of vertices. It is important to note that shortest paths may not be unique, 
meaning that a given problem instance might have more than one valid solution. 


Definition 7.12 A decision problem is any problem to which the answer is simply yes or 
no (or, more formally, 1 or 0). 


Examples of decision problems are: Is this proposition satisfiable? Is this graph Eulerian? 
Is this graph Hamiltonian? And so on. 

Numerous problems fall into the category of optimization problems rather than decision 
problems. These optimization problems’ involve identifying the optimal solution, which 
could be either the maximum or minimum value, from a set of feasible solutions. Feasible 
solutions are essentially sets of values for the decision variables that satisfy all the con- 
straints laid out in the optimization problem. For instance, consider the shortest path prob- 
lem, where the goal is to discover a feasible path with the fewest edges possible. Notably, 
NP-completeness pertains directly to decision problems, not optimization problems. Nev- 
ertheless, it is typically possible to reformulate an optimization problem as an equivalent 
decision problem by introducing a constraint on the optimized value. To illustrate, we can 
frame a decision problem associated with the shortest path problem as follows: Given an 
undirected graph G, two vertices u and v within G, and a positive integer k, does there exist 
a path between u and v that comprises at most k edges? 

In order to introduce the NP-complete problems, we must first define the class NP. Before 
that, we define the following notions and terminology in the computational complexity 
theory. Recall that an efficient algorithm is the one that can run in polynomial time. We 
also have the following definitions. 


Definition 7.13 A problem is said to be tractable if there exists an efficient algorithm that 
solves all instances of it, and is intractable otherwise. 


Definition 7.14 A certificate of a solution to a decision problem is a scheme that is used 
to check and certify whether a decision problem gives the answer yes or no. 


Definition 7.15 A decision problem is called verifiable if a certificate of a solution to the 
problem exists. 


In particular, a decision problem is verifiable in polynomial time if for every instance with 
the answer yes, there is a solution S which we can use to check in polynomial time that the 
answer is yes. 


5 The shortest path problem will be studied in Subsection 9.3.2. 
6 Eulerian and Hamiltonian graphs were defined in Section 4.3. 
7 Optimization problems will be formally studied in Part IV. 


77 The Complexity Class NP-Complete 
The notion of reduction from one problem to another explains to us how problems relate. 


Definition 7.16 A reduction of a decision problem Q, to a decision problem Q, is a map- 
ping of every instance q, of the problem Q, to an instance q, of the problem Q, such that 
q, is yes if and only q, is yes. 


A reduction provides us with a way to state, “If I can find a solution for problem Q,, then I 
can also find a solution for problem Q,.” In such a scenario, if, for instance, Q, is solvable in 
polynomial time, it implies that Q, can also be solved within polynomial time. This insight 
helps us gauge the relative difficulty of problem Q,. A notable property of reductions is their 
transitive nature. Specifically, if we can demonstrate that Q, reduces to Q, in polynomial 
time, and Q, reduces to Q, in polynomial time, then we can infer that Q, reduces to Q, in 
polynomial time. 


Definition 7.17 A problem is classified as P if it can be solved in polynomial time. 


All of the actual problems we have thus far solved in this book have been in P. Addi- 
tionally, many problems in the book, including all of the problems we will study in 
Chapters 8-12, are in P. 

Because P stands for polynomial, many people assume that NP must stand for 
“nonpolynomial.” This is incorrect. The set NP refers to “non-deterministic polynomial.” 
We need some more definitions. 


Definition 7.18 A problem is classified as NP if it can be verified in polynomial time. 
Note that P C NP. In fact, being able to solve a problem in polynomial ensures that we 
can also verify the problem by simply solving it. Figure 7.17 shows a Venn diagram of the 


different classes of problems. 


Definition 7.19 A problem is classified as NP-hard if any NP problem can be reduced to 
this problem in polynomial time. 


NP-complete NP-complete 


IfP CNP If P= NP 


Figure 7.17 Graphical relationships among the complexity classes P, NP, NP-hard, and 
NP-complete. 
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In other words, a problem is classified as NP-hard if an algorithm for solving it can be 
translated to solve any NP problem. In light of Definition 7.19, an NP-hard problem is at 
least as hard as any NP problem (it could be even much harder). 

NP-complete problems are problems that reside in both the NP and NP-hard classes. 


Definition 7.20 A problem is classified as NP-complete ifit can be verified in polynomial 
time and that any NP problem can be reduced to this problem in polynomial time. 


Examples of NP-Complete problems include the Hamiltonian path problem, the Boolean 
satisfiability problem, and the integer programming problem. 

It is a natural question to ask how these sets relate to each other. A million-dollar question 
is: Does P=NP? This is a well-known unsolved problem in mathematics and theoretical 
computer science. 


Exercises 
7.1 Choose the correct answer for each of the following multiple-choice questions/items. 


(a) Which loop is guaranteed to execute at least one time? 

(i) For loop. (ii) While loop. (iii) Do-while loop. (iv) Not any listed. 

(b) What is the use of for loop? 

(i) To repeat the statement a finite number of times. 

(ii) To repeat the statement until any condition holds true. 
(iii) To repeat the statements for infinite time. 
(iv) To repeat statements inside until any condition is false. 

(c) We have three algorithms to solve a problem, Algorithms 1, 2, and 3. The run- 
ning time function of Algorithm 2 is the function g(n) = n?, that of Algorithm 1 
is the function f(n) = 22 e(n), and that of Algorithm 3 is the function 
h(n) = zgn. Here, n is the input size. Which algorithm is asymptotically 
the fastest? 

G) Algorithm 1. Gi) Algorithm 2. (iii) Algorithm 3. 
(iv) All algorithms have the same performance. 

(d) We have three algorithms to solve a problem. The running time function of Algo- 

rithm 1 is f(n) = 1000n, that of Algorithm 2 is g(n) = n?, and that of Algorithm 3 

ish(n) = = n>, where nis the input size. Which algorithm is asymptotically the 

fastest? 
(i) Algorithm 1. (ii) Algorithm 2. (iii) Algorithm 3. 

(iv) All algorithms have the same performance because when n = 1000, which 

is a large input size, f(1000) = g(1000) = h(1000) = 10°. 

Suppose that we have an algorithm of seven lines and the cost of execution each 

line is a constant, say c. By doing a line-by-line analysis, we found that the run- 

ning time of the algorithm is c[(n — n®)/(1 — n)]. How many times line i executes 

(fori =1,2,...,7)? 


(e 


s 
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(i) ni-times. Gi) n&)-times. (iii) n“-times. (iv) Not any listed. 


(f) In the fragment given in Algorithm 7.15, if the for-statement in line (2) is 
replaced with for (i=1; i<=n; i++), and that in line (3) is replaced with 
with for (j=1; j<=n; j++), then what is the new running time? 

(i) The running time will not change. 

(ii) Itisc, + en +e (n -— 1)n + e(n — 1}. 
(iii) Itis c; +c (n + 2) +c (n + 1)(n + 2) +0¢,(n + 1}. 
(iv) Itisc +c (n +1) +6 (n — 1)(n + 1) + c&(n — 1). 

(g) Suppose that we have an algorithm of five lines, the cost of execution each line 
is the constant c, and the line i executes n! times for each i = 1, 2, ... , 5. If a line- 
by-line runtime analysis is used, what is the Po time of the algorithm? 


~ e(n- n°) e(n- n°) ca-n) .. (1 —n°) 
(i) l-n (ii) Iicn Gi ji = (iv) n 
(h) Letf(n) be the running time function of the aa given in Algorithm 7.30. Which 


one of the ee summations represents f(n)? 


© fy = y y Ye (iii) fn) = DDN 


a i=1 j=i k=i 

n-l n n-l n j 
(ii) f(n) = DI (iv) f(n) = PÈ ye 

i=1 j=i k=i i=l j=i+1 ksi 


Algorithm 7.30: The algorithm of Exercise 7.1 (h) 
1:x=0 
2: for (i = 1;i < n;i + +) do 
3: | for(j=i+1;j<n;j++)do 


4: for (k = i;k < n;k + +) do 

5: | x=x+(i-j—k); // Execution cost is c 
6: end 

7. | end 

8: end 

9: return x 


Algorithm 7.31: The algorithm of Exercise 7.1(i) 
1:x=0 
2: for (i = 1;i < n;i + +) do 
3: | for (j = logn;j > 1;j— -—) do 
4 | x=x+(i-j); // Execution cost is c(i, j) 
5 | end 
6: end 
7: return x 
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(i) Given the program snippet in Algorithm 7.31. Which one of the following sum- 


mations represents the running time of this program? 
n logn n logn 


© >.> cep. (iii) $ $ cG logn -j + D. 
i=1 j=l i=1 j=l 
n logn (iv) None of the above. 
(ii) x > c(i, logn — j). 
i=1 j=l 


(j) If the running time of an algorithm is a polynomial in n and is represented 
by the sum Yi, Dad + + D+ UG +2)+---+(@-1) +7), which one of the 
following correctly describes the algorithm’s running time? 

(i) quadratic. (ii) cubic. (iii) quartic. (iv) quintic. 

(k) Which of the following asymptotic statements is/are true? 

(i) n?+n=O(n?). (iii) n? +n = O(n’). 
(ii) n? +n = O(n’). (iv) None of the above. 

(1) Let f(n) = O(g(n)) and k < 0. Which of the following asymptotic statements 

is/are true? 


(i) IfI = Ogn). (iii) [kf (n)| = Q(g(n)). 
(ii) kf] = Oen). (iv) All of the above. 
(m) Which one of the following statements about Theorem 7.1 is incorrect? 
(i) The converse of item (i) is false. (iii) The converse of item (ii) is false. 
(ii) The converse of item (i) is true. (iv) The contrapositive of (ii) is true. 
(n) Which one of the following is true? 
(i) a" is O(b") if1 < b <a. (iii) n* is O(b") for any a and b > 1. 
(ii) ntis O(n’) if1 <b <a. (iv) a” is O(n?) for any b and a > 1. 


(0) Let f(n) be the worst-case running time function of the code given in 
Algorithm 7.32. Which one of the following summations represents f (n)? 


64 64 64 64 8 4 
(i) +> se (iii) Ds 
i=0 j=0 k=1 i=0 j=0 k=0 
64 8 4 63 7 3 
(ii) ye (iv) ye 
i=0 j=0 k=1 i=0 j=0 k=1 


(p) If the running time of an algorithm is a polynomial of degree 5 in n and is rep- 
resented by the sum 
Y Hk + K+ 1) + (k + 2) + cdots + (m? — 1) + m?)(640)]. 
m=1k=1 
Which of the following is the greatest lower bound? for the algorithm’s running 
time? 
(i) =n’. (ii) =n. (iii) $n’. (iv) $n’. 


8 By the greatest lower bound, we mean the largest polynomial bounds the running time from below. 


Exercises 
Algorithm 7.32: The algorithm of Exercise 7.1 (0) 
1: cnt = 0 
2: for (i = 0; i < 64; i + +) do 
3: | for (j = 0;j? < 64;j ++) do 
4: for (k = 1; k? < 64/k;k ++) do 
5 if (2i +j > 3k) then 
6 | cnt ++; // Execution cost is c 
7: end 
8: end 
9: | end 
10: end 
11: return x 
(q) Which one of the following is false? 
(i) ntis O(n?) ifa < b. (iii) a” is O(b") if1 < b < a. 
(ii) n? is not O(n?) if b < a. (iv) a” is not O(b") if1 <b <a. 
(r) Let f(n) = (n? — 1)°. Which of the following is true about f (n)? 
(i) fn) = O(n’). (iii) f(n) = O(n’). 
(ii) f(n) = O(n’). Gv) fin) = O(n"). 
(s) The tight Big-Oh bound is best expressed by 
(i) O. (iii) ©. 
(ii) Q. (iv) All the above. 


(t) Which one of the following functions is a simple and tight bound on the function 
fn) = (5.01)""1 + 5"+!? 
(i) 6.0). (ii) (5.01)". (iii) 5”. (iv) S, 
(u) Which one of the following functions is a simple and tight bound on the function 
f(n) = (9.99)"*1 + (10)" + (10.01)""!? 
(i) (9.99)"*1. (ii) (10)". (iii) (10.01)""1. (iv) (10.01)". 
(v) Which one of the following is/are true? 
(i) If a logarithmic-time algorithm exists to solve a problem, then we can say 
that the problem is solved efficiently. 
(ii) If a polynomial-time algorithm exists to solve a problem, then we can say 
that the problem is solved efficiently. 
(iii) If a polylogarithmic-time algorithm exists to solve a problem, then we can 
say that the problem is solved efficiently. 
(iv) All the above. 


7.2 Write an algorithmic code of the following procedure which solves the same problem 
in Example 7.6. 
(i) If Lis of length 1, return the first item of L. 
(ii) Set v1 to the first item of L. 
(iii) Set v2 to the output of performing find-max () on the rest of L. 
(iv) If v1 is larger than v2, return v1. Otherwise, return v2. 


231 


232 


7 Analysis of Algorithms 


7.3 


7.4 


7.5 


For the algorithm obtained in Exercise 7.2, answer the following questions by “Yes” 
or “No” with any necessary comments. 

(a) Does it have defined inputs? (c) Is it guaranteed to terminate? 

(b) Does it have defined outputs? (d) Does it produce the correct result? 


Find the running time equipped with worst-case performance for Algorithm 7.12 
by carrying out a line-by-line analysis. Assume that the cost of the statement in line 
(i) isc fori = 1,2,3,4,7. 


Do a line-by-line analysis to find the running time of the fragment given in 
Algorithm 7.33. 


Algorithm 7.33: The algorithm of Exercise 7.5 


1: sum = 0; // Cost = c) 
2: for (i = logn; i > 1; i — —) do // Cost = c, 
3: | for (j= n;j > 1;j/ = 2) do // Cost = c, 
4: | sum + = array[i][j]; // Cost = c, 
5 | end 
6: end 

7.6 Let p and q be two positive integers such that p < q. How many times do we go 


7.7 


7.8 


around each of the following loops, as a function of p, q, and possibly k? (Here k is a 
positive integer. Also, in item (f), we assume that q is divisible by k and its powers). 


(a) for (i = p;i < q;i + +). (d) for (i = q;i > p;i- =k). 
(b) for (i = q;i > p;i — —). (e) for (i = p;i < q;i* = k). 
(c) for (i = p;i < q;i+ = k). (f) for (i = q;i > p;i/ = k). 


Given the code in Algorithm 7.34, write the summations that represent the running 
time of the code and solve the summations to find an approximate running time. 
You must show all your work without doing a line-by-line analysis. 


Given Algorithm 7.35, write the summations that represent the running time of the 
algorithm and solve them to determine an upper bound. You must show all your 
work without doing a line-by-line analysis. 


Algorithm 7.34: The algorithm of Exercise 7.7 


ty =0 
2: for (i = n?%; i < n? + 5;i + +) do 
3: | for (j= 4;j < n;j+ +) do 


4: | x=x+(i-j); // Execution cost is c 
5 | end 
6: end 


7: return x 


Exercises 


Algorithm 7.35: The algorithm of Exercise 7.8 


1::x=0 


: for (i = 1;i < n;i + +) do 


for (j = 1;j < 3i; j + +) do 
| x=x+(i-j); // Execution cost is c 
end 


6: end 


7: return x 


7.9 


7.10 


7.14 


7.16 


Find upper and lower bounds for the running time of the algorithm represented 
3 3 

by the following summation f(n) = Pye pa c, where c is the execution cost of the 

statement that contributed most to the running time of the algorithm. 


Given Algorithm 7.30 which has a running time function f(n). Use bounding to solve 
the summations that represent f(n) and find a tight asymptotic bound for f(n). You 
must show all your work. 


Use the Big-Oh definition to prove the following asymptotic statements. 
(a) 5n? — 3n + 20 = O(n’). (c) 5n? — 4n — 2n? + n = O(n’). 
(b) 4n? — 12n + 10 = O(n?). (d) n? + ynsinn + nlogn = O(n’). 


Use the Big-Omega definition to prove the following asymptotic statements. 


(a) 4? +n +1 = Qn. (b) nlogn — 2n + 13 = Q(n logn). 


Use the Big-Theta definition to prove the following asymptotic statements. 


(a) nñ +n? +7n +1 = O(n). (b) in? -= 3n = O(n?). 


Let fis fa» --- Sie Si Bo» +--+ -8g be asymptotically nonnegative functions. Assume that 
fi(n) = O(g,(n)) for each i = 1, 2, ...,k, prove that 
k 


k k 
(a) Di fi) = O (max j cx {8i(n)})? (>) [[ fa =0 (I so)” 
i=1 i=1 i=1 
Prove Properties 7.1 and 7.6. Prove also Properties 7.2 and 7.7 for Big-Oh. 


Use limits to prove the following asymptotic statements. 


(a) Vinz +1=0(n).  @) n” =Q). (c) yYn+4- yn = 00). 


(d) nlog (n?) + (n — 1} log (5) = © (n? logn)." 


9 It is also true that ELAN =0 oe gi(n)). 


10 Recall that ELAN) = fi(n) + f(n) +- -+ f(n). Likewise, JE HENAC) = fihan) - f(n). 
11 Hint: For (b), you may use the equation: n log(n?) + (n — 1} log(n/2) = 2n logn + (n — 1) (logn — log 2). 
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7.17 


7.18 


Let f(n) = 2“"*!, Does f(n) = O(2")? Answer by YES or NO. If yes, use the Big-Oh 
definition to show that f(n) is O(2”). If no, use the proof by contradiction to show 
that f(n) is not O(2”). 


Choose the correct answer for each of the following multiple-choice questions/items. 

(a) Consider the selection statement given in Algorithm 7.2. IfS, and S, are O(f,(n)) 
and O(f,(n)), respectively, and the conditionis"2 == V2 ", which can tell 
which branch of the selection statement is taken, then the running time of the 
selection statement is 


Gi) of o). Gii) O(f,(n)). 
(ii) OA +f). (iv) O(max {f,(1),()})- 

(b) Consider the while statement given in Algorithm 7.6. If the statement (s) is 
(are) O(f(n)), and the conditionis "1 != Vi", which is known to be true 
or false right from the start, then the running time of the while statement is 

(i) oa). (iii) OC. + f(n)). 
Gi) O(n). (iv) None of the above. 


(c) Given the program snippet in Algorithm 7.36. What is the time complexity of 
this program? 
(i) O((n?/2) log(n)). (iii) O(n? log(n)). 
(ii) O(nlog?(n)). (iv) O((n? /2)log?(n)). 
(d) Given the program snippet in Algorithm 7.37. What is the time complexity of 
this program? 
(i) O(n? + mlog(m)). (iii) O(n? + m?). 
(ii) O(n? + log(m)). (iv) O(n? + m/2). 


Algorithm 7.36: The algorithm of Exercise 7.18 (c) 
1: for (i = n;i > 1;i/ = 2) do 


2; 
3: 
4: 


for (j = 1;j < log(i);j + +) do 
| something O(n) 
end 


5: end 


Algorithm 7.37: The algorithm of Exercise 7.18 (d) 


1: something O(n + m) 
2: while (n > 0) do 


something O(n) 
h _— 


5: end 
6: while (m > 0) do 


something O(m) 
m/ =2 


end 


7.19 


7.20 


7.21 


7.22 


Exercises 


Consider the fragment in Algorithm 7.26. Note that f(n) is a function call. Give a 
simple and tight Big-Oh upper bound on the running time of Algorithm 7.26, as a 
function of n, on each of the assumptions below. Justify your answers with complete 
evidences. 

(a) The running time of f(n) is O(n), and the value of f(n) is n. 

(b) The running time of f(n) is O(n), and the value of f(n) is n!. 


Suppose that line (17) of Algorithm 7.27 is replaced with “for (i = 1;i < cat(n, n); 
i + +)” What would the running time of “main” be then? Justify your answer with 
complete evidence. 


Algorithm 7.38 is called a stochastic!” interior-point algorithm and is used for 
solving a class of two-stage stochastic linear optimization problems of K scenarios 
and with n variables in the first stage and m variables in the second stage. The 
input includes a matrix A and a barrier function f, and the output is an e-optimal 
solution to the given stochastic linear program. No matter how the algorithm works 
and no matter how the mathematical notations therein look like and how they 
work, the graph structure (flowchart) of the algorithm can be visualized. Draw the 
graph structure for Algorithm 7.38 by completing the subgraph diagram shown in 
Figure 7.18. 


Consider Algorithm 7.38. Suppose that the running time of the block in lines (5)-(7) 
is O(m), and the running time of the block in lines (9)-(12) is O(n). Similarly, sup- 
pose that the running time of the block in lines (17)-(19) is O(m), that of the block 
in lines (21)-(24) is O(n), and that of the statement in line (29) is O(m). Suppose 
also that all other statements (including those of checking the conditions of the 
while loops) are of constant times O(1). Let also N;,, be the number of times we go 
around the inner while loop of line (13). In view of the graph structure obtained in 
Exercise 7.21, determine the running time of the algorithm as a whole in each of 
these cases: 

(a) Nin = O(n + Km) and y € (0, 1) is an arbitrarily chosen constant. 

(b) Na = O(1) and y = 1 — o / yn + Km, where o > 0. 

In each case, give your answer as a function of n, m, K, „® and e using Big-Oh, and 
write a paragraph (or more) that justifies your determination. 


12 Deterministic problems do not consider any uncertainties in input data, whereas stochastic problems 
model the uncertainties in data with appropriate probability distributions. 
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Algorithm 7.38: The algorithm of Exercise 7.21 

1: initialize e > 0,7 € (0,1),0 > 0,8 > 0.x, pW > 0,4 
2: set x ê x®, pS pO, A £ 40 

3: while (u > e) do 

4. | for(k=1;k <K+1;k++)do 


5: find (yY, z®, s®) 

6: choose a scaling element p 

compute ( y”, a by scaling (y®, s®) with p 
s: | end 


9: | compute g = Vf(u, x)— ATA 

compute Ax = — (Gas = (Vf) AT (A (vf) AT) A 07) (2) 
te. =: = 

i: | compute Ad = (4 (Vf) AT) A (Vf) @) 


ix | Compute 5(u, x) = 4/ ADTT NAD 


13: | while (ô > f) do 


14: set x Ê x + 0Ax 
15: set 4 4+0A4 
16: for (k = 1;k < K + 1;k + +) do 
17: find (y®, z®, s®) 
18: choose a scaling element p 
ja: compute ( y”, s) by scaling (y®, s®) with p 
20: end 
21: compute g = Vf(u,x)— ATA 
wo. | | compute Ax = - (Pn - (Vf) AT (A (Vf) AT) A 027) (8) 
-1 = -1 
ia compute AA = (4 (Vf (4, x)) Av) A(V2f(u.x)) e) 
compute 6(u, x) = 4/ LCAs? f(y, x)(Ax) 
: H 
25: | end 


26: | setu yu 
27: end 
28: for (k = 1;k < K + 1;k + +) do 


29; | apply inverse scaling to ( w, a to compute (y, s®) 


30: end 
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Figure 7.18 A subgraph structure of 
Algorithm 7.38. 


Begin algorithm 


initialize e, y, 0, B, x, u™,A©, f,A 


xxO, u 4 HOA = A) 
y afai 
- ——| Bs u = €? | ———_> .... 
yes} 
ket 


| 


[is k< K+1?|-—> >. --. 


yes | 


find (y®, z®, s) 


k++ 


Notes and Sources 


The history of algorithms and their analysis is intertwined with the development of math- 
ematics and computing. As mentioned in the introductory paragraphs of this chapter, the 
term “algorithm” itself derives from the name of Al-Khwarizmi, who lived in the 9th century 
and made significant contributions to mathematics. Al-Khwarizmi’s work laid the founda- 
tion for algebra and the systematic solving of linear and quadratic equations, essentially 
providing a structured approach or algorithm for these problems (refer to Khuwarizmi- 
et al. (1963)). In the 20th century, the field of algorithm analysis saw substantial growth 
with the development of computer science. One of the key references in this field is Donald 
Knuth’s multivolume work Knuth (1997), which delves into the history and analysis of 
algorithms. 

This chapter delved into the essential concepts of algorithmic analysis. By exploring 
asymptotic notation, readers gained a deep understanding of how to quantify and assess 
the running time of algorithms. This knowledge enabled us to evaluate and compare 
algorithm efficiency, ultimately aiding in the selection of efficient algorithms for finding 
more optimized solutions in mathematics and computer science. 
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As we conclude this chapter, it is worth noting that the cited references and others, such 
as Cormen et al. (2001); Rosen (2002); Cusack and Santos (2021); Mott et al. (1986); Liu 
(1968); Grimaldi (1999); Joshi (1989); Maurer and Ralston (2004); Hougardy and Vygen 
(2016); Higham (2002); Kagaris and Tragoudas (2008); Greene and Knuth (2009); Lewis 
and Papadimitriou (1998), and Sipser (2013), also serve as valuable sources of information 
pertaining to the subject matter covered in this chapter. Exercise 7.22 is due to (Zhao, 2001, 
Section 5). 
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Standard array and numeric algorithms, such as array multiplication algorithms, array 
searching algorithms, array sorting algorithms, and Newton’s method algorithm for root 
finding are all discussed in this chapter. We also discuss the integer Euclidean algorithm 
(or Euclid’s algorithm), which is one of the oldest and simplest number theoretic 
algorithms. Now we start with array multiplication algorithms. 


8.1 Array Multiplication Algorithms 
Vectors and matrices are arrays of real numbers. In this section, we will take a look at some 


simple matrix computations. Namely: Multiplying a matrix by a vector, and multiplying a 
matrix by a matrix. 


8.1.1 Matrix-Vector Multiplication 


Consider a matrix with n rows and m columns, that is, an n x m matrix: 


Ay) Ayn Qiz --- Qim 
Aa a ae aas se ane 
any an2 an3 78 Anm 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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We assume that the entries of A are real. Given an m-tuple of real numbers: 


we can multiply A by x to get a product b = Ax, where b is an n-tuple. The ith component 
of b is obtained by taking the dot product of ith row of A with x. That is, its ith component 
is given by 


b; = Fao. (8.1) 


Example 8.1 An example of matrix-vector multiplication with n x m = 3 x 2 is 


12 7 1X7+2x8 23 
34 Ke 3X7+4X8]=] 53}. 
5 6 


5x7+6X8 77 - 


A computer code to perform matrix-vector multiplication looks something like the one 
in Algorithm 8.1. 
Algorithm 8.1: Matrix-vector multiplication (row-oriented) 
1:b=0 
2: for (i = 1;i < n;i + +) do 
3: | for (j= 1;j < m;j + +) do 


4: | b; =b; + aX; 
5: | end 
6: end 


Note that in Example 8.1, we have 


23 1 2 
53|=7 |3|+8 |4]. 
77 5 6 


In general, if b = Ax, then b is a linear combination of the columns of A. So, if a; denotes 
the jth column of A, then we have 


j=l 
Expressing these operations as a code, we have the one in Algorithm 8.2. 
If each vector operation is performed by a loop, the code becomes the one in 
Algorithm 8.3. 
Please take note that Algorithms 8.1 and 8.3 are essentially the same, with the only differ- 
ence being the interchange of loops. Algorithm 8.1 accesses A by rows, while Algorithm 8.3 
accesses A by columns. 
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Algorithm 8.2: Writing b = yin a,x; in a code 


1:b=0 

2: for (j = 1;j < m;j + +) do 
3: | b=bt+ax, 

4: end 


Algorithm 8.3: Matrix—vector multiplication (column-oriented) 
1::b5=0 
2: for (j = 1;j < m;j + +) do 
3: | for (i = 1;i < n;i + +) do 


4: | b, =b; + aX; 
5: | end 
6: end 


Algorithm 8.4: Matrix—matrix multiplication 
1: B=0 
2: for (i = 1;i < n;i + +) do 
3: | for(j=1;j < lj++)do 


4: for (k = 1;k < m; k + +) do 
5: | by = by + AX 

6: end 

7. | end 

8: end 


Now, our objective is to determine the time required for the completion of Algorithm 8.3. 
It is important to note that each iteration of the inner loop involves two floating-point oper- 
ations (flops). To simplify our analysis, we can represent each loop using a summation 
symbol x. Given that the inner loop executes for values of i ranging from 1 to n, and there are 
two flops performed in each iteration, the total number of flops executed during a single run 
of the inner loop can be expressed as $; 2. Considering the outer loop, which runs from 
j = 1 toj = m, we can calculate the overall number of flops as X4; Di1 2 = Èj- 2n = 2nm. 
This means that a total of 2nm flops are performed during the execution of Algorithm 8.3. 


8.1.2 Matrix—Matrix Multiplication 


If A is an n x m matrix, and X is an m x l matrix, we can form the product B = AX, which 
is n X l. The (i,j) entry of B is the dot product of the ith row of A with the jth column of X. 
That is, the (i,j) entry of Bis 


m 
by = Vary: (8.2) 
k=l 


A computer code to perform matrix—matrix multiplication looks something like the one in 
Algorithm 8.4. 
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Note that if l= 1, the matrix—-matrix multiplication represented in (8.2) reduces to 
the vector-matrix multiplication represented in (8.1), and Algorithm 8.4 reduces to 
Algorithm 8.1. 

Now, we want to know how long it will take to complete the task of Algorithm 8.4. Note 
that every iteration of the innermost loop includes two flops. Replacing each loop with a 
summation sign È, the total number of flops is 


n l m 
£ £ $2 = 2nml. 
i=1 j=1 k=1 
In the important case when all of the matrices are square of dimension n x n. We end this 
section by summarizing the above computational complexities in terms of n, where n is the 
size of the square matrix; see Corollary 8.1. 


Corollary 8.1 The time complexities of the above matrix algorithms are as follow: 


(i) The matrix-vector multiplication algorithms are quadratic, running in O(n?) time. 
(ii) The matrix-matrix multiplication algorithm is cubic, running in O(n*) time. 


8.2 Array Searching Algorithms 


Searching algorithms are a class of algorithms that finds an element within an array or list. 
In this section, we introduce and analyze two well-known searching algorithms, namely 
linear search and binary search. 


8.2.1 Linear Search 


Linear search, also called sequential search, is a linear-time searching algorithm that 
sequentially checks each element of the list to be searched until a target is found. The 
linear search is stated in Algorithm 8.5, in which we search an array A[0 : n — 1] to find 
a value x that is expected to be present in the array. If the value x is not present in the 
array, the algorithm terminates unsuccessfully. To arrive at the Big-Oh of the running time 
for Algorithm 8.5, we depict a flowchart of this algorithm with running time complexity 
in Figure 8.1. It is evident that the while loop encompassing lines (3) to (5) within 
Algorithm 8.5 can be executed up to a maximum of n times, but not beyond that. Applying 
the summation rule, we can deduce that the time complexity of Algorithm 8.5 is O(n), as 
this represents the maximum time among the following components: the time required for 
the assignment in line (2), the time for the while loop in lines (3)-(5), and the time for the 
selection statement in lines (6)-(11). 

The running time found here (which is O(n)) is the worst-case performance of 
Algorithm 8.5. What are its best- and average-case performances? This question is left as 
an exercise for the reader (see Exercise 8.1 (c)). 


8.2 Array Searching Algorithms 


O((1\(n)) = O(n) O(n) 


At most 
n-times 


around 


O (max {1, 1})= O(1) 


return x 
not found 


return 
index i 


O(1) oO(1) 


Figure 8.1 Linear search graph structure with running time complexity. 


Algorithm 8.5: Linear search algorithm 
Input: An array A[0 : n — 1] anda sought value x 
Output: Index i such that x = A[i], otherwise x is not found 

1: linear-search(int A[ ], int n, int x) 

zi=0 

3: while (x! = A[i] && i < n) do 

4: i++; 

5: end 

6: if (i < n) then 

7. | printf(“xis present at index” i) 

8: end 

9: else 

10: | print £(“x is not present in the array”) 


11: end 
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Algorithm 8.6: Binary search algorithm 
Input: An array A[low:high] of length n and a sought value x 
Output: Index i such that x = A[i], otherwise x is not found 
1: binary-search(int A[ ], int low, int high, int x) 
2: if (low > high) then 


3: | print £(“x is not present in array”) // constant time c, 
4: end 

5: else 

6: | mid = |(low+high)/2| // constant time c, 
7. | if (x = A[mid]) then 

8: | print £(“x is present at index” mid) // constant time c, 
9: | end 

10: | if (x< A[mid]) then 

11: | binary-search(A,low, mid-1, x) // same problem of size n/2 
12: | end 

13: | if (x > A[mid]) then 

14: | binary-search(A, mid+1, high, x) // same problem of size n/2 
1s: | end 

16: end 


8.2.2 Binary Search 


This Subsection introduces another searching algorithm, called binary search. It is worth 
mentioning that binary search (and merge sort that we studied in Subsection 8.3.3) are well- 
known examples ofa class of algorithms called divide and conquer algorithms. Within this 
class of algorithms, the approach involves breaking down the problem into “simplified” 
iterations of itself. These simplified problems are then conquered using the same proce- 
dure, typically in a recursive manner, and the outcomes of these “simplified” versions are 
amalgamated to construct the ultimate solution. 

Binary search, also known as half-interval search, or logarithmic search, is a searching 
algorithm that finds the position of a target within a sorted array A[low : high]. The binary 
search algorithm is stated in Algorithm 8.6. 

For more illustration of the binary search algorithm, in Table 8.1 we show in detail how 
to apply Algorithm 8.6 to an eight-element array. We first target the value x = 8, which is 
present in the array at index 6. Then, we also target x = 6 which is not found in the array. 

Let T(n) be the running time of Algorithm 8.6 on an array of size n. From Algorithm 8.6, 
it follows that the recurrence formula for this recursive program is 


T) = c, 


T(n) = c + T(n/2), (8.3) 


where c might be taken equal to the constant time c, + c, + c,. From Example 5.4, we have 


T(n) = ck + T(n/25) = clogn + T(1) = clogn + c. 


8.2 Array Searching Algorithms 


Table 8.1 A concrete implementation of the binary search algorithm. 


Input: A = [1 23 45 8 9 10], 
x= 8. 


Iteration #1: 
low high 


| mid 4 
1 3 4 5 
1 3)-4)5 )8) 9 | 10 


2 
2 


Iteration #2: 

As x = 8 > A[mid] = 4, we get 
low high 
| mid | 
I G@ T P 


1 2) a ESS 9 | 10 


Iteration #3: 
As x = 8 = A[mid], we stop! 


Output: x is present in A at index 6. 


Input: A = [1 23 45 8 9 10], 


x= 6. 
Iteration #1: 
low high 
| mid L 
1.2 3.4. >) 6 7 8 
12/31415] 8/9 10 
Iteration #2: 


As x = 6 > A [mid] = 4, we get 
low high 
| mid | 
S 6 7 8 

121/3141518] 9 10 


Iteration #3: 
As x = 6 < A [mid] = 8, we get 
low mid high 
Ndg 
5 


1/2;),3/4)5}]8)9 4 10 


Iteration #4: 
As x = 6 > A[mid] = 5, we get 
high low 
te al 
5 6 
1/2});3/4)5}]8)9 4 10 


Iteration #5: 
As low > high, we stop! 


Output: x is not present in the array A. 


Thus, the recurrence formula (8.3) of Algorithm 8.6 describes the logarithmic func- 
tion T(n) = clogn +c. As a result, the binary search algorithm is logarithmic, running 
in Odogn) time. This is the worst-case performance of Algorithm 8.6. What are its 
best- and average-case performances? This question is left as an exercise for the reader 


(see Exercise 8.1 (e)). 


We end this section with the following corollary which summarizes the computational 
complexity (worst behavior) in terms of the size of the list (n). 
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Corollary 8.2 The (worst-case) time complexities of the above searching algorithms are 
as follow: 


(i) The linear search algorithm is linear, running in O(n) time. 
(ii) The binary search algorithm is logarithmic, running in O(log n) time. 


8.3 Array Sorting Algorithms 


Sorting algorithms are algorithms that put elements of an array or list in a certain order. 
In this section, we introduce and analyze three well-known sorting algorithms, namely 
insertion sort, selection sort, and merge sort. 


8.3.1 Insertion Sort 


Insertion sort is a sorting technique that resembles the way many individuals organize a 
hand of playing cards: 


e Initiate with a single card in your hand. 

e Select the subsequent card (to be sorted), create space for it by shifting the already sorted 
items, and place it in its appropriate position. 

e Continue the above step for all the cards in the hand. 


Figure 8.2 illustrates the insertion sort algorithm by applying the above steps to sort the 
array [57 32 41 60 11 33]. Note that, at each iteration, the array is divided into two sub- 
arrays. A formal description of the insertion sort method is given in Algorithm 8.7. 


Algorithm 8.7: Insertion sort algorithm 
Input: An integer array A[0 : n — 1] 
Output: Array A sorted in ascending order 

1: insertion-sort(int A[ ], int n) 


2: for (i = 1;i < n;i + +) do // Cost = c4, # times = n 
3: | next = A[i] // Cost = c,, # times =n — 1 
4. | j=i-1 // Cost = c}, # times =n — 1 
5: | while (j > 0 && A[j] > next) do // Cost = c,, # times = D t, 
6: Allj +1] = Afi // Cost = cs, # times = Y(t; — 1) 
7: j=j-1 // Cost = ce, # times = P (t — 1) 
8: | end 

9: | Aff +1] = next // Cost = c}, # times =n — 1 
10: end 


Note that “next” in line (3) is the item to be sorted, line (6) shifts sorted items to make 
room for “next,” and line (9) inserts “next” to the correct location. 


8.3 Array Sorting Algorithms 


Partially sorted Unsorted 


@) 


@) 
@) 
@) 


©) 
©) 
© 
© 


Figure 8.2 A concrete implementation of the insertion sort algorithm. 


In the side comments in gray in Algorithm 8.7, t; denotes the number of times the while 


statement is executed at iteration i for i = 1, 2, ...,n — 1. Given this, the total cost is 
n-1 
fin) = cyn+c,(n—1)+0,(n—-1)+ ei, 
n-i n-1 g (8.4) 


+05 Y(t; — 1) +e ih -— + e,(n- 
i=1 i=1 


We now present best- and worst-case analyses for the insertion sort method: 


e Best-case analysis. The array is already sorted. So, A[j] < next upon the first time the 
while-loop test is run, hence t; = 1 for each i = 1,2, ..., n — 1. It follows that 


fn) = (ci +c, +c, +c, +c) — (C, +0, +e, +c) = en +e. 


tin I < ee 
¢ (say) é (say) 


Thus, when best-case analysis is considered, the rate of growth for the insertion sort 
method is n. 
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e Worst-case analysis. The array is sorted in reverse order. So, we always have A[j] > 
next in while-loop test. Then we have to compare “next” with all elements to the left of 


the ith-position. That is, we compare with i — 1 element. So, t; = ifori=1,2,...,n—1. 
Using (8.4) and applying the arithmetic series formula (see Corollary 2.1), it follows that 
fan) = an+ e(n- 1)+c(n-— 1)+ cn- 1) 


n-1 n-1 n-1 
+e, Pt +e WG -D +e V(G-D 
i=1 i=1 i=1 


cyn+c,(n—1)+c,(n—1)+c,(n—1) 


+a (1) +e (222) +e (22d) 


= cn? + ĉn + Č, 


ll 


for some constants c,¢ and č. Thus, when worst-case analysis is considered, the rate of 
growth of the insertion sort algorithm is n?. 


8.3.2 Selection Sort 


Selection sort is a sorting method that partitions the input list into two segments: a sorted 
sublist, which gradually forms on the right-hand side, and an unsorted sublist containing 
the remaining items on the left-hand side. At the outset, the sorted sublist is devoid of ele- 
ments, while the unsorted sublist encompasses the entire input list. The algorithm advances 
by identifying the smallest element within the unsorted sublist, swapping it with the left- 
most unsorted element, and shifting the boundaries of the sublist one element to the right. 

For more illustration of this method, we apply the selection sort method to sort the 5- 
element list (21, 35, 22, 32, 74). Table 8.2 shows all steps in this method for sorting the given 
list. The selection sort is formally stated in Algorithm 8.8. 

To arrive at the Big-Oh of the running time for Algorithm 8.8, we represent the structure 
of this algorithm and the running time complexity by the tree structure shown in Figure 8.3. 

In closing, the running time of selection sort is O(n”). This is the worst-case performance 
of Algorithm 8.8. What are its best- and average-case performances? This question is left as 
an exercise for the reader (see Exercise 8.1 (c)). 


Table 8.2 A concrete implementation of the selection sort algorithm. 


Least element in 


Unsorted sublist the unsorted list Sorted sublist 
(21, 35, 22, 32, 74) 11 () 

(35, 22, 32, 74) 12 (21) 

(35, 32, 74) 32 (21, 22) 

(35, 74) 35 (21, 22, 32) 
(74) 74 (21, 22, 32, 35) 


() (21, 22, 32, 35, 74) 


(n — 1)- 
times 


around 


(n-i- 1)- 
times 


around 


O(1) 


Figure 8.3 Selection sort tree structure with running time complexity. 


Algorithm 8.8: Selection sort algorithm 
Input: An integer array A[0 : n — 1] 


O(n -i- 1) 
or O(n) 


Output: Array A sorted in ascending order 


1: selection-sort(int A[ ], int n) 


2: for (i = 0;i < n—1;i++)do 
small =i 


4 | for(j =i+1;j<n;j++)do 
5: if (A[j] < A[small]) then 
6: | small = j 

7: end 

8: | end 

9: | swap = A[small] 


10: | A[small] = A[i] 
1: | A[i] = swap 


12: end 


8.3 Array Sorting Algorithms 


O(n) 


O(n? — n) 
or O(n?) 
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Algorithm 8.9: Merge sort algorithm 
Input: An integer array A[first:last] of length n 
Output: Array A sorted in ascending order 
1: merge-sort(int A[ ], int first, int last) 
2: if (first < last) then 


3: | mid = |(first+last)/2 | // Constant time c 
4. | merge-sort(A, first, mid) // Same problem of size n/2 
5 | merge-sort(A, mid+l, last) // Same problem of size n/2 
6: | merge(A, first, mid, last) // A call to “merge” takes time n 
7: end 


8.3.3 Merge Sort 


Merge sort is a sorting algorithm in which the input is an unsorted array A[first : last], and 
the output is the array A sorted in ascending order. The idea behind the merge sort method 
involves breaking down the unsorted list or array, consisting of n elements, into n sorted 
sublists, with each sublist initially containing a single element (where a single-element list 
is regarded as a sorted array). Subsequently, these sublists are systematically merged to gen- 
erate fresh sorted sublists, and this process continues until only one sublist remains. That 
final sublist represents the sorted array. The merge sort algorithm is stated in Algorithm 8.9. 

Note that Algorithm 8.9 uses the “merge” function, which is an inbuilt function in 
C++STL! that merges two sorted lists into a single sorted list. 

For more illustration of the merge sort algorithm, in Figure 8.4 we show in detail how to 
apply Algorithm 8.9 to a seven-element array. 

Let T(n) be the running time of Algorithm 8.9 on an array of size n. From Algorithm 8.9, 
by dropping the constant time c in favor of the large time n, it follows that the recurrence 
formula for this recursive program is 


T(n) = n + 2T(n/2). (8.5) 
From Example 5.5, we have 
T(n) = nlogn + nT(1). 


Thus, the recurrence formula (8.5) of Algorithm 8.9 describes the linearithmic function 
T(n) = n(logn + constant time). As a result, the merge sort algorithm is linearithmic, run- 
ning in O(n logn) time. This is the worst-case performance of Algorithm 8.9. What are its 
best- and average-case performances? This question is left as an exercise for the reader (see 
Exercise 8.1 (e)). 

We end this section with the following corollary which summarizes the computational 
complexity (worst behavior) in terms of the size of the list (n). 


Corollary 8.3 The (worst-case) time complexities of the above sorting algorithms are as 
follow: 


(i) The insertion sort algorithm is quadratic, running in O(n?) time. 


1 C++STL stands for the C++ Standard Template Library. 


8.4 Euclid’s Algorithm 


48 || 37 | 53 | 13 || 19 || 92 || 20 


48 || 37 | 53 || 13 | < |__| 19 || 92 | 20 
48 | 37 | < 53 | 13 19 | 92 | <— ‘> | 20 
43 |= == | 37 53. | S is 19 | <— * | 92 = | 20 


37 || 48 13 || 53 19 || 92 20 


13 || 19 || 20 || 37 || 48 || 53 || 92 


Figure 8.4 A concrete implementation of the merge sort algorithm. 


(ii) The selection sort algorithm is quadratic, running in O(n?) time. 
(iii) The merge sort algorithm is linearithmic, running in O(n logn) time. 


Besides the above sorting algorithms, there are many other sorting algorithms that are not 
presented in this book. This includes heap sort, quicksort and counting sort (see Cormen 
et al. (2001)). In practical implementations a few algorithms predominate. Insertion sort is 
widely used for small data sets, while for large data sets an asymptotically efficient sort is 
used, primarily merge sort, heap sort, or quicksort. 


8.4 Euclid’s Algorithm 


The greatest common divisor of two integers is the largest number that can evenly divide 
both integers without leaving a remainder. An effective approach for calculating the greatest 
common divisor of two integers is the Euclidean algorithm, also known as Euclid’s algo- 
rithm. This algorithm was originally formulated by Euclid around 300 BCE and can be 
found in Book VII of his Elements. 

There are two versions of Euclid’s algorithm: The division-based version which recur- 
sively calls a function while the larger number is not divisible by the smaller, and the 
subtraction-based version which recursively subtracts the smaller number from the larger. 
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The greatest common divisor of integers a, b € N is written as gcd(a, b). We write a modn 
to denote the unique integer b such that 0 < b < nanda = b (mod n) (i.e., the remainder ofa 
when divided by n). For example, 39 = 3 (mod 12) because 38 — 3 = 36, which is a multiple 
of 12. It is known that gcd (a, b) = b when b divides a. In addition, if a is not divisible by b, 
the reader can show that gcd (a, b) = gcd (b, a mod b) (see Exercise 8.5). This leads us to the 
correctness of Algorithm 8.10. 

For example, the gcd(1377, 594) is calculated from the equivalent gcd(594, 1377 mod 594) 
= gcd(594, 189). The latter greatest common divisor is calculated from the gcd(189, 462 mod 
189) = gcd(189, 27), which in turn is calculated from the gcd(21, 189 mod 27) = gcd(27, 0) 
= 27. Thus, gcd(1377, 594) = 27. 

Note that, in Algorithm 8.10, if negative inputs are allowed, then the mod function may 
return negative values. In this case, the instruction “return a” must be replaced with “return 
max (a, —a).” 

Contrary to the recursive version stated in Algorithm 8.10, which works with two 
arbitrary integers as input, the subtraction-based version works with two positive integers 
and stops when they are both identical. This version of Euclid’s Algorithm is stated in 
Algorithm 8.11. 

For illustration, the reader can use Algorithm 8.11 to check that gcd(1377, 594) = 27. 

Let us analyze the worst-case time complexity of Algorithm 8.11 with respect to the 
sum, denoted as n = a + b, of a and b. Excluding our base case, this sum decreases with 
each recursive step. Considering that the smallest possible reduction in each step is 1 
(for instance, gcd(x,1)), and since every positive integer is assured to have a common 
divisor of 1, we can conclude that the time complexity is O(n). Consequently, the number 
of iterations in Algorithm 8.11 can grow linearly in relation to the sum of a and b. 

We now estimate the worst-case time complexity of Algorithm 8.10 based on the 
minimum, n = min (a, b), of a and b. Note that the number of iterations Algorithm 8.10 
will take is maximized when the two inputs are consecutive Fibonacci numbers. As 
introduced in Section 3.1, the Fibonacci numbers fo» f1» f2» ..., are defined by the recurrence 
relation 


fa = faa tfai- for n = 2,3,4,..., where fp =0 and fi =1. 


Using this recurrence formula, we have f, = 1,f, = 2,f, =3,f, =5,f, =8.f, = 13, etc. 
More specifically, if a > b > 0 and gcd(a, b) requires n > 1 steps, then we can verify that 
the smallest possible values of a and b are f,,,, and f,,,, respectively (see Exercise 8.6). 
Therefore, b > f,,,. 


Algorithm 8.10: Euclid’s Algorithm (the division-based version) 
1: gcd(a, b) 
2: if b = 0 then 
3: | return a 
4: end 
5: else 
6: | return gcd(b, a mod b) 
7: end 


8.5 Newton's Method Algorithm 


Algorithm 8.11: Euclid’s Algorithm (the subtraction-based version) 
1: gcd(a, b) 


2: while (a 4 b) do 
3: | if (a> b) then 
4: | aĉa-b 

5: | end 

6: | else 

z | | b&b-a 
s: | end 

9: end 

10: return a 


Exercise 5.6 uses the generating function method to solve the Fibonacci recurrence f, = 
Sn—-1 + fna-2- One can find (see the solution to Exercise 5.6 in Appendix A) that 


1 
= —(a"-(1-a)"), 
Í RL a= a)") 


where, for n > 0, « ê (1 + 5) /2 is the “golden ratio.” A more simplified version of the 
above formula is the following: 


It follows that 


b > fari x I yy = pa = V5 ina) = aD, 
24/5 
Solving for n, we get 


n < log,b + 1 = O(log,n). 


Thus, the number ofiterations in Algorithm 8.10 can be logarithmic based on the minimum 
ofa and b. 


8.5 Newton’s Method Algorithm 


Newton’s method (also called Newton-Raphson method) is one of the root-finding algo- 
rithms. Given a function f, the goal of a root-finding algorithm is to find a root of f, or 
solution of the equation f(x) = 0. The desired value r that satisfies f(r) = 0 is also called a 
zero of the function f. 

Let f : [a,b] —> R be a twice continuously differentiable function. Let rọ € [a, b] be 
an approximation to a root r of f such that f'(r,) #0 and |r — rọ| is small. Consider the 
following Taylor expansion for f(x) about rọ evaluated at x = r. 

(r— Gan 


2 3 
w F" Cro) + Co y geese 


0 =fr) = fo) + (r — rof’ fo) + 6 
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y=f@, 


Tangent line y = f(ro) + (x -— ro)f (ro) 


Figure 8.5 The geometry of Newton’s method. 


Newton’s method can be derived by assuming that the terms involving (r — ry)*, (F — rọ)’, -.. 
are tiny because |r — ro| is small. So 


0 & f(r) + (r — rof" (ro). 
After solving for r, we immediately have 


rar fro) =r. 
O 
Figure 8.5 shows the geometry of Newton’s method. The tangent line to the curve 
y=f(x), namely y = f(r) + Œ — rọ f'(ro), intersects the x-axis at r; and has the slope 
of f'(rọ). Initializing at the approximation ry, the x-intercept of the tangent line to the 
graph of f at (ro. f(ro)), Say r}, becomes a better approximation to r. Re-initializing at the 
approximation r,, the x-intercept of the tangent line to the graph of f at (r,,f(r,)), becomes 
even better approximation than r, to r, and so on. 
Accordingly, Newton’s method initializes with an initial point r, and generates a sequence 
of points {r;,}/°, using 


reas =e AE, for aa ee (8.6) 


The following example was taken from Burden and Faires (2010). 


Example 8.2 Approximate a root of the function f(x) = cosx — x using Newton’s Method 
and starting with r) = 2/4 = 0.7853981635. 


Solution 
Equation (8.6) for our setting becomes 


COS Ty — Fk 


Koy = Te = 
ko k sinr- 


for k = 0,1,2, ... . This gives the approximations in Table 8.3. E 


8.5 Newton's Method Algorithm 


Table 8.3 Numerical results of 


Example 8.2. 
k That 
0 0.7395361337 
1 0.7390851781 
2 0.7390851332 
3 0.7390851332 


Algorithm 8.12: Newton’s method algorithm 
Input: Initial guess rọ, maximum number of iterations N, tolerance TOL 
Output: Approximate solution r to f(x) = 0 or message of failure 

1: setk =0 

2: while (k < N) do 

3: | compute r,,, using (8.6) 

4 | if [Fk — rk] < TOL then 


5: print £(“r equals” r;,,,) // The procedure is successful 
6: break 

7 | end 

8: | setk ++ 

9: | Set rg = Fk 
10: end 
11: print £(“Fails after N iterations”) // The procedure is unsuccessful 


Newton’s method is formally stated in Algorithm 8.12. 
We give, without proof, the following convergence and complexity result in this context. 
For a proof, see Cheney and Kincaid (2007). 


Theorem 8.1 Let f : [a,b] —— R be a twice continuously differentiable function. 
If r € (a,b) is such that f(r) =0 and f’(r) #0, then there exists 6 > 0 such that New- 
ton’s method generates a sequence {r,}°° , converging quadratically to r for any initial 
approximation satisfying |r — ro| < 6. 


8.5.1 Newton’s Method for Nonlinear Systems 


Newton’s method can be also used to find roots of nonlinear equations. In general, given a 
system of equations when the number of unknowns equals the number of equations. We 
are interested to find solutions to a system of the form: 


fi, Oy Xo3-.-5X,) = 0, 
fai iee) 


ll 
S 


(8.7) 


ll 
S 


AeA e 
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where f, : R” —— R fori=1,2,...,n. In vector form, System (8.7) can be written as 
f(x) = 0, (8.8) 


where f = (f,;f,;....f,,) : Rt — R” and x Ê (x,;x,;...;X,) E€ R”. 
Newton’s method can be extended to the nonlinear system (8.8) using 


rth = rf _ (Fe FV len 


where J,f is the n x n Jacobian matrix defined as 


Sh (x) h(x)... Lexy 
hoy By. Bagg | [Cee 
ax, ©) ax pii on, ©) _ CAAC 


li> 


If 
(V afn 00 


dfn fa (x)... Pa 
ix, ©) x, ©) ae, (x) 


and, forafunctiong : R” > R, Vg(x) is the gradient vector (the vector of the first derivatives 
of g) defined as 


I> 
Q 
S 


Vg) (8.9) 


Note that the method presumes that the Jacobian Jf is nonsingular at each r, soits inverse 
exists. The two-step procedure in the following workflow, followed by an example, will 
teach us how to solve System (8.8). 


Workflow 8.1 We solve System (8.8) by following two steps: 
(i) Solve 7,f Ar = —f(x) for Ar. (ii) Let re! = r* + Ar. 


The vector Ar defined in Workflow 8.1 is called a Newton direction. 


Example 8.3 Let us perform one iteration of Newton’s method to approximate the solu- 
tion of the following set of equations with a starting guess rọ = (1/2, 1/2)". 


xi +x, =l, 
-x +33 =-1. 
Letting 
fx) = o 2 Ever 
fa X3) -x +x +1)’ 
we have 
2x? 1 
= 1 
as =e a 
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Then 
_ [-3/8 _ [3/4 1 -1 _ 16 [3/4 -1 
fro) = | 5/8 Ie fro) = k A , and hence [J fro] = 35| 1 3/4" 
Performing one iteration of Newton’s method, we get the following Newton direction 
_ ai _ _16 [3/4 -1] [-3/8] _ [29/50 
Ar = AIS $0) = -35 | 1 3/4] | 5/8] [22/50] ` 


As a result, we have 


_ [1/2] [29/50] _ [27/25 
ot ne = fA a = Ka i 


which is the approximation after the first iterate. Note that, with a sharp eye, the reader can 
see that there is one and only one solution to f(x) = 0, namely (x,,x,) = (1, 0). E 


8.5.2 Newton’s Method for Optimization 


In this part, Newton’s method is exploited to optimize twice differentiable convex functions. 
Consider the function g : R” ——> R, which is a real-valued function of n independent 
variables. We say that g is differentiable on a domain D C R” if V,g exists for all x € D, 
where V,g is the gradient vector defined in (8.9). Similarly, g is twice differentiable on D 
if V2 g exists for all x € D, where V2,g is the Hessian matrix (the matrix of the second 
derivatives of g) defined as 


ag og arg 
ox? (x) OX, OX, (x) OX, OX, (x) 
ag og arg 
2 _ a | Axx, (x) ax; (x) OX, 0X, (x) 
Vix = (8.10) 
ag ag ag 
OX, OX, (x) OX, OX, (x) ax? (x) 


Note that the Hessian of a function is the Jacobian of its gradient. That is, 
Virg = Iy V x8 


In addition, g is called twice continuously differentiable on D if V2,g is continuous on D. 
As a matter of fact, if a function is a twice continuously differentiable function, then its 
Hessian matrix is symmetric. Moreover, if a function is a twice continuously differentiable 
convex function, then its Hessian matrix is positive semidefinite (see Definition 3.3). 
Assume that we are interested in minimizing a twice continuously differentiable real val- 
ued function g : R” ——> R. That is, we are interested in a problem of the form: 
min g(x). (8.11) 


xER" 
In this case, we want to equate its gradient to the zero vector. That is, we need to find the 
roots of the system of nonlinear equations 


vV gŒ) = 0. (8.12) 
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Algorithm 8.13: Newton’s algorithm for unconstrained optimization 
Input: Initial guess x°, maximum number of iterations N, tolerance TOL 
Output: Approximate optimal solution to Problem (8.11) 

1: setk =0 

2: while (k < N) do 

3: | compute Ax = —[V2,9(x*)]-1V_g(x*) 

4: | choose step-size 6, 

5: | set xt! = xk + 6, Ax 

6: | if ||x,4, — x,|| < TOL then 


7: print £(“Approximate optimal solution is” x***) 
8: break 
9: | end 

10: | setk ++ 

u: | set x1 = xk 

12: end 


13: print £(“The method fails after N iterations”) 


Note that, in (8.12), the number of unknowns equals the number of equations. Applying 
Newton’s method here involves the Jacobian matrix of V,g, which is its Hessian matrix 
defined in (8.10). 

Algorithm 8.13 is used to minimize the function g. In Algorithm 8.13, the vector Ax is 
called a Newton direction or Newton step, x* is called the Newton iterate, and the parameter 
0, is called the step-size. Also, in Algorithm 8.13, ||-|| denotes the Euclidean norm. 

Example 8.4, which was taken from Freund (2004), performs one iteration of 
Algorithm 8.13 to get an approximation of the optimal solution of a minimization problem. 


Example 8.4 With a starting guess x° = (0.85; 0.05), let us perform one iteration of 
Algorithm 8.13 to approximate the optimal solution of the problem: 


min ln — x, —x,) -lnx -lnx 
nen 17% 1 2 


where D = {(x,;x,) € R? : x > 0,x, > 0,x, +x, < 1}. Letting 


g(x) = —In(1— x, — x,) -lnx — lnx, 
we have 
1 i 
|4747 
V 8x) = i a 
1-xX,-xX, xX, 
and 
1 1 1 
(l-x,-x,)? x? (1-x,- x)? 
Vg) = 1 l 1 1 


8.5 Newton's Method Algorithm 
Then 


Vga’) x pa EEA ne | l 


100 500 


Performing one iteration of Newton’s method, we get the following Newton direction 


Ax = -[V2,8(x*)]-!V,g(x") 
VORRE! 500 —100 | [8.824 
~ 40100 | —100 100.2 | | —10 


_ [-0.012468 0.002493 | [8.824] _ [-0.134947 
~ | 0.002493 —0.002498| | —10 | | 0.046978 |` 


As a result, we have 


0.8 —0.134947 0.7150 
ene - pa 


0.05 0.046978 0.096978 


which is the approximation after the first iterate. The step size was chosen to be 6, = 1. 

As an exercise, the reader is encouraged to perform more iterations to find x* for 
k = 2,3,4, 5. The approximate optimal solution after the fifth iteration is x° = (0.333338; 
0.333259). It is not hard to check that the optimal solution (x,,x,) = (1/3, 1/3). E 


The Newton direction (or generally, any search direction) is called a decent direction if 
g(x + OAx) < g(x) for all sufficiently small values of 6. In general, without some further 
assumptions, there is no guarantee of decent directions in Algorithm 8.13. We have the 
following theorem. 


Theorem 8.2 Ifg : R” —— R is a twice continuously differentiable strictly convex real 
valued function, then the Newton direction Ax is descent. 


Proof: If g is strictly convex, then its Hessian matrix, V2,,g(x), is positive definite (the proof 
for this fact is left as an exercise for the reader). It follows that 


(V(x) "Ax = —(V,8(%))"[VZ.8(0)1 V g) <0, 


where the inequality follows from the fact that [V2,g(x)]"! is positive definite (see 
Exercise 3.7). 
Now, g can be approximated by its first-order Taylor expansion at x: 


g(x + 0Ax) x g(x) + O(V g)" Ax < g(x). 
Thus, Ax is a descent direction. E 


It can be shown that Algorithm 8.13 has a quadratic convergence under some certain 
circumstances (see, e.g., Freund (2004)). 
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Exercises 


8.1 


8.2 


Choose the correct answer for each of the following multiple-choice questions/items. 

(a) Let n be a positive integer. The total number of flops when we multiply an n? x n 
matrix with an n-tuple of real numbers is 
(i) 2n. (ii) 2n?. (iii) 2n3. (iv) 2n. 

(b) Letn bea positive integer. IfA and Bare n x n? and n? x n? matrices, respectively, 
then the total number of flops after multiplying A with B is 


(i) 2n3. (ii) 2n*. (iii) 2n°. (iv) 2n°. 

(c) The best-case time complexities of linear search and selection sort are respectively 
(i) O(1) and O(). (iii) O(1) and O(n?). 
(ii) O() and O(n). (iv) O(n) and O(n?). 


(d) If the function main in Algorithm 8.14 calls not only the function karger but 
also the function selection-sort that we analyzed in Algorithm 8.8. What is 
the worst-case time complexity of the new resulting program? 


(i) O(mn + n?). (iii) O(mn + m?). 
(ii) O(m? + n?). (iv) O(mn). 
(e) The best-case time complexities of binary search and merge sort are respectively 
(i) O() and O(1). (iii) O(1) and O(n logn). 
(ii) O() and O(n). (iv) Odogn) and O(n logn). 


(f) Which of the following sorting algorithms, if any, is an example of an algorithm 
for which the best- and worst-case time complexities differ? 


(i) Merge sort. (iv) None of the above because, in 
(ii) Insertion sort. all sorting algorithms, the best- 
(iii) Selection sort. and worst-case time complexities 


always coincide. 
(g) Which of the following, if any, is an example of a searching algorithm for which 
the best- and worst-case time complexities coincide? 
(i) Linear search. 
(ii) Selection sort. 
(iii) No such algorithm exists because the best- and worst-case time complexities 
always differ. 
(iv) None of the above. 
(h) Using Algorithm 8.10, we find that gcd(55, 99) = 


(i) 9. (ii) 10. (iii) 11. (iv) 12. 
(i) Using Algorithm 8.11, we find that gcd(840, 3220) = 
(i) 120. (ii) 140. (iii) 160. (iv) 180. 


When we perform the multiplication of two matrices, both of size n xn, it typi- 
cally involves 2n? flops. However, in 1969, Strassen introduced a novel approach, 
documented in Strassen (1969), which accomplishes this task using an algorithm 
that scales with O(n‘) flops, where the exponent s is approximately equal to 2.81 
(computed using log,7). Given that 2.81 is less than 3, Strassen’s method proves 
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8.4 


8.5 


8.6 


8.7 


Exercises 


to be more efficient than Algorithm 8.4 when the value of n is sufficiently large. 
Empirical tests have revealed that Strassen’s method can even outperform the 
conventional approach when n is relatively small, such as around 100. Nevertheless, 
because the difference between 2.81 and 3 is relatively small, we need to increase 
the value of n substantially before Strassen’s method demonstrates a significantly 
superior performance. Moreover, it’s worth noting that there exist even faster matrix 
multiplication techniques. For instance, Coppersmith and Winograd introduced 
a method, as discussed in Higham (2002), that can multiply two nxn matrices 
in approximately O(n?3’°) flops. However, despite this impressive reduction in 
complexity, the computing community has observed that this algorithm does not 
outperform Strassen’s method. How can we explain this observation? 


What is the worst-case time complexity of the program shown in Algorithm 8.14? 
Justify your answer. 


What is the worst-case time complexity of the program shown in Algorithm 8.15? 
Justify your answer. 


Prove that gcd (a, b) = gcd (a mod b, b) for any a,b EN. 


Consider Algorithm 8.10. Assume that a > b > 0 and that gcd(a,b) requires n > 1 
steps. Show that the smallest possible values of a and b are f, and f,,,,, respectively, 
where {fa}; o is the Fibonacci sequence defined as 


Ía =Snratfn—2 for n = 2,3,4,..., with f =Oandf, =1. 


Write a code to find a root of the function f(x) = ix? +x+1-— e on [0,4]. Apply 
Newton’s method starting with x) = 1. 


Algorithm 8.14: The algorithm of Exercise 8.1(d) and 8.3 


1: # include <stdio. h > 


2: 
3: 


int karger(intx, int m) 

int linear-search(int A[ ], int n, int x) 
// This line is left intentionally blank 

: main() begin 


int A[0 : n-—1],m 
a=karger(0, m) 
print £(“Value a: \ n”) 
return a 


: end 
// This line is left intentionally blank 
: karger(int x, int m) begin 


for (i = 1;i < m;i + +) do 

| x+= linear-search(A[ ],n,m) 
end 

return x 


: end 
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Algorithm 8.15: The algorithm of Exercise 8.4 


1: #include <stdio. h > 
2: int dinic(int x, int m) 
3: int binary-search(int A[ ], int low, int high, int x) 
4: int merge-sort(int A[ ], int first, int last) 
// This line is left intentionally blank 
5: main( ) begin 
6: | intA[O:n—1],m 
7z | A[n—1] = dinic(O, m) 
s: | B=merge-sort(A[ ],0,n—-1) 
9: | print £(“Sorted array: \ n”) 
10: | return B 
11: end 
// This line is left intentionally blank 
12: dinic(intx, int m) begin 
13: | for (i=1;i< m;ix = 2) do 
14: | x += binary-search(A[ ],0,n — 1, m) 
15: | end 
16: return x 
17: end 


Notes and Sources 


The origins of the numeric and array algorithms date back to ancient times. Euclid’s algo- 
rithm, developed by the ancient Greek mathematician Euclid in the third century BCE, 
is one of the earliest known numeric algorithms (see, for example, Heath et al. (1952)). 
Newton’s method, named after Sir Isaac Newton in the 17th century, is also a historical 
numeric algorithm that laid the groundwork for the development of numerical analysis 
(refer to Newton (1687)). When it comes to array searching and sorting algorithms, the 
history is also rich. One of the earliest sorting algorithms, known as the “bubble sort,” was 
described in 1956 by John Mauchly. Additionally, John von Neumann’s work on merge sort- 
ing algorithms in the 1940s significantly contributed to the development of array sorting 
techniques (refer to Von Neumann and Burks (1966)). 

This chapter described and analyzed fundamental algorithms that form the core of com- 
puter science and numerical analysis. Specifically, it covered various searching and sorting 
algorithms, including linear search, binary search, insertion sort, selection sort, and merge 
sort, which play a crucial role in data manipulation and organization. The chapter also 
delved into specific numerical methods, such as Euclid’s algorithm for greatest common 
divisor calculations and Newton’s method for solving linear and nonlinear systems. 

As we conclude this chapter, it is worth noting that the cited references and others, such as 
Cusack and Santos (2021); Mott et al. (1986); Grimaldi (1999); Maurer and Ralston (2004); 


References 


Hougardy and Vygen (2016); Kagaris and Tragoudas (2008); Greene and Knuth (2009); 
Sipser (2013), and Watkins (1991), also serve as valuable sources of information pertaining 
to the subject matter covered in this chapter. Exercise 8.2 is due to Watkins (1991). 
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The graph-searching algorithms are the main focus of this chapter. Before studying graph- 
searching algorithms, it is crucial to first acquire an understanding of graph representa- 
tion. This foundational knowledge is essential as it forms the basis for employing these 
algorithms effectively. Exploring the techniques for graph representation becomes the cor- 
nerstone of our initial section, serving as a fundamental stepping stone leading to the sub- 
sequent investigation and exploration of search algorithms. 


9.1 Graph Representations 


There are two standard ways to represent a graph, namely the adjacency list and the 
adjacency matrix. They both can be used for directed and undirected graphs. 


9.1.1 The Adjacency List Representation 


An adjacency list is a collection of unordered lists used to represent a (directed or undi- 
rected) graph. We write an array of |V| lists. Each list describes the set of neighbors of a ver- 
tex in the graph. The list that describes the set of neighbors ofa vertex u ina graph G = (V, E) 
is denoted by Adj[u] and is defined to contain all vertices v that are adjacent u. That is, 


Adj[u] ê {v E€ V : (u,v) € E}. 
The following example illustrates this. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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Example 9.1 Use adjacency lists to represent each of the following graphs. 


(a) The directed (b) The undirected (c) The undirected 
graph: graph: graph: 


Solution 
(a) Note that Adj[1] = {3}, Adj[2] = {1,3}, and Adj[3] = {2}. So, the adjacency list repre- 
sentation for the graph is: 


2 e > 1/@ >3/¢e x] 


(b) Note that Adj[1] = {2,5}, Adj[2] = {1, 5, 3, 4}, Adj[3] = {2,4}, Adj[4] = {2, 5,3}, and 
Adj[5] = {4,1,2}. So, the adjacency list representation for the graph is: 


1 e >+2/@e@ +5 /e >| 

2 Le lle >5\e 13) @ >4le AX] 
3 e >2\¢e > 4 |© >| 

4 Le 12] 15 le 13] NX] 

5 e 14/e 1le 12] x] 


(c) This part is left as an exercise for the reader (see Exercise 9.3 (a)). a 


Note that, for an undirected graph, in the adjacency list there are two representations of 
each edge in the graph. 


9.1.2 The Adjacency Matrix Representation 


An adjacency matrix is a square binary matrix! used to represent a (directed or undirected) 
graph. We assume that the vertices of the given graph G = (V, E) are numbered 1,2, ..., |V|. 
The representation consists of a (0, 1)-matrix Ajy,)y,, where the entry of the ith row and jth 


1 A binary matrix is a matrix with entries from the set {0,1}. 


9.1 Graph Representations 


column in the matrix is labeled a, and is given by 


A { 1, if(@ij) €E, 


ï ) 0, otherwise. 


The following example illustrates this. 
Example 9.2 Use adjacency matrices to represent each of the following graphs. 


(a) The undirected (b) The directed (c) The undirected 
graph: graph: graph: 


PD 
i) LO 

ONS 

o © 


The adjacency matrices that represent the graphs in items (a) and (b) are as follow: 


112]/3]41516 

1/12/3]4]5 1);0;1);0}1/0)0 
1/0;1;0;0;1 2/0);0];/0;0}]1)0 
2/1/;0;1];1)1 3/0)/0)/0;0; 1) 1 
3/0])1);0}]1/0 4/0/1/0/0])0]0 
4/0/;1/;1)/0)]1 51010/011010 
S5/1])1)0}1/0 6/0|;0])/0;0;0)} 1 

(a) (b) 
(c) This part is left as an exercise for the reader (see Exercise 9.3 (b)). E 


It is worth noting that in the case of a simple graph, the adjacency matrix exhibits a 
specific characteristic: All the diagonal elements of the matrix are zero. This is because, 
in a simple graph, no vertex is connected to itself directly, so the diagonal entries of the 
adjacency matrix must be zero. Note also that each edge in the graph is represented 
twice. This is due to the fact that in an undirected graph, the connection between two 
vertices is mutual. Consequently, the adjacency matrix of an undirected graph turns out 
to be symmetric, as it contains mirrored entries above and below the main diagonal. 
This symmetry further underscores the fundamental difference between directed and 
undirected graphs when analyzing and manipulating graph data represented by adjacency 
matrices. 

We end this section with Table 9.1, which shows some differences between the adjacency 
list representation and the adjacency matrix representation. From Table 9.1, we can see 
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Table 9.1 Comparison between the adjacency list and adjacency matrix representations of a given 
graph G = (V, E). 


Adjacency list Adjacency matrix 
Space complexity: O(V + E) space O(V?) space 
Time to list all vertices adjacent to u € V: O(deg(u)) time O(V) time 
Time to determine if an edge (u, v) € E: O(deg(u)) time O(1) time 
Is it efficient for sparse matrices? Yes No 
Is it efficient for dense matrices? No Yes 
Is it a quick way to list the vertices Yes No 
adjacent to a given vertex? 
Is it a quick way to determine if there is an No Yes 


edge between two vertices? 


that the choice between these representations depends on the graph’s characteristics and 
the specific operations or algorithms being applied. 


9.2 Breadth-First Search Algorithm 


Breadth-first search is a method for exploring a graph. In this section, we describe and ana- 
lyze this method. In Section 9.3, we will describe some of its applications. 

In a breadth-first search, we explore outward from a source vertex, say s, of a graph G = 
(V, E) in all possible directions, adding vertices one layer (or level) at a time. Breadth-first 
search is used for both directed and undirected graphs. Figure 9.1 shows the breadth-first 
search scheme, where L, represents the zeroth move (which only includes the a source 
vertex S), L, represents the first move, L, represents the second move, etc. 

In Figure 9.1, Lo £ {s}, L, £ Adj[s] (i.e., the neighbors of L,), L, contains all vertices that 
do not belong L, or L} and that have an edge to a vertex in L,, and L,,, contains all vertices 
that do not belong to an earlier layer, which is needed to avoid duplication, and that have 
an edge to a vertex in L;. We have the following remark. 


Remark 9.1 For each i, the layer L; consists of all vertices at distance exactly i from the 
source vertex s. 


Figure 9.1 The breadth-first search scheme. 


9.2 Breadth-First Search Algorithm 
The following example explains the steps outlined in the breadth-first search method. 


Example 9.3 For each of the following graphs, use the breadth-first search method 
to determine the smallest number of layers (or hopes) among the vertices starting from 
vertex s. 


(a) The undirected graph in the left-hand side of Figure 9.2 (take s to be the vertex 1). 
(b) The directed graph in the right-hand side of in Figure 9.2. 


2nd move: 


Figure 9.3 Illustrating the progress of breadth-first search on the graph of Example 9.3 (a). 
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Solution 

(a) In this part, the breadth-first search method is explained move-by-move as shown in 
Figure 9.3. We find that the smallest number of layers among the vertices starting from 
the vertex s (vertex 1) equals 4. 

(b) As shown in Figure 9.4, the smallest number of layers among the vertices starting from 
the vertex s equals 6. a 


Ls Figure 9.4 Applying the breadth-first search 
m —, onthe digraph of Example 9.3 (b). 
l 


Algorithm 9.1: Breadth-first search algorithm 
1: BFS(s, Adj) 
2: begin 


3: | level= {s : 0} 

4: | parent = {s : None} 

5: i=1 

6: | frontier = [s] 

7: | while frontier do 

8: next = [ ] 

9: for u in frontier do 

10: for v in Adj[u] do 

11: if v not in level then 
12: level[v] =i 

13: parent[v] =u 
14: next.append(v) 
15: end 

16: end 

17: end 

18: frontier = next 

19: it=1 

20: | end 

21: | return parent 


22: end 


A version of a breadth-first search algorithm is presented in Algorithm 9.1. 
Theorem 9.1 gives the running time of Algorithm 9.1 on an input graph G = (V, E) given 
in its adjacency lists. 


Theorem 9.1 Algorithm 9.1 runs in O(V + E) time. 


9.3 Applications of Breadth-First Search 


Proof: Note that, on an input graph G = (V, E) given in its adjacency lists, we have at most 
|V| layers (lists) and each vertex occurs on at most one layer and is captured at most once. 
So, the while-loop and the first for-loop in Algorithm 9.1 are executed at most |V| times. 
Then we have O(V). The second for-loop in Algorithm 9.1 runs |F,,;| times, where E,aj is 
the set of edges that are incident to current vertex. The reason is that every vertex captured 
at most once and we examine (u, v) only when u is captured. Note that every edge examined 
at most once if Gis directed, and at most twice if G is undirected. Therefore, from Theorems 
4.1 and 4.14, |V| x O(E,4;) is O(E). The other lines in Algorithm 9.1 take constant time to 
execute. Thus, the total running time is |V| x (O(1) + O(E nj) which turns out to O(V + E). 
This completes the proof. a 


From Theorem 9.1, we conclude that the breadth-first search algorithm runs in time lin- 
ear in size of the adjacency list representation of G. 


9.3 Applications of Breadth-First Search 


The breadth-first search has some applications, and there are some O(V + E)-time 
algorithms based on the breadth-first search algorithm for the following problems: 
Computing a spanning tree (forest) of a graph, computing a path with the minimum 
number of edges between start vertex and assigned vertex or reporting that no such path 
exists, testing bipartiteness of an input graph, testing whether a graph is connected, and 
finding the connected component that contains a given vertex. Below we describe these 
applications. 


9.3.1 Computing Spanning Trees (Forests) 


For a given graph G = (V, E), the breadth-first search algorithm inherently creates a 
spanning tree that has its root at the source vertex. If the given graph G is disconnected, 
then the breadth-first search produces a spanning forest of G. The tree (forest) produced 
by performing a breadth-first search is called breadth-first tree (forest), and its edges are 
called breadth-first edges. 

For the given graph shown to the left of Figure 9.5, we compute a spanning tree by draw- 
ing the produced breadth-first tree. As shown to the right of Figure 9.5, the breadth-first 


src © 


Figure 9.5 A graph and its breadth-first tree. 
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edges are colored blue and the vertices are enumerated in alphabetical order starting at a 

(the src vertex), where each vertex is also super-indexed by (i) to mean that it occurs on the 

layer L; for some i. As breadth-first search does not define the order of the neighbors, we 

assume that in this graph the children of each vertex are ordered from top to bottom. 
Figure 9.5 leads us to deduce the following remark. 


Remark 9.2 If (u,v) is a breadth-first edge, then the levels of u and v differ by at most 1. 


9.3.2 Computing Shortest Paths 


Finding shortest paths is one of the applications of the breadth-first search algorithm. 
The shortest path problem is a well-known problem in graph theory and it has numerous 
applications in the real world. It involves finding a path from a source vertex to a destina- 
tion vertex which has the minimum number of edges (i.e., with the least length) among all 
such paths, or report that no such path exists. The following remark gives evidence when 
such existence is affirmed. 


Remark 9.3 There is a path from a (source) vertex s to a vertex t if and only if t appears 
in some layer while performing the breadth-first search algorithm. 


The procedure in following workflow, followed by an example taken from Section 22.2 in 
Cormen et al. (2001) with some modifications, will show us how to find the shortest path(s) 
in an input graph G given by its adjacency representation. 


Workflow 9.1 We find the shortest path(s) from a source vertex to a destination vertex t 
in an input graph G by following two steps: 


(i) Run a breadth-first search starting at the vertex s. 
(ii) Construct a path whose edges are all from breadth-first tree edges by moving backward 
from vertex t to its predecessors along the tree edges until we reach vertex s. 


From Workflow 9.1, we conclude that 
(S, ... , parent[parent[t]], parent[t], t) 


is a shortest path from vertex s to vertex t. We also conclude that the length of this path 
equals level[t]. That is, if vertex t occurs on a layer L, (i.e., level k), then the length of the 
above shortest path is k. 


Example 9.4 As shown Figure 9.6, by performing the breadth-first search algorithm, the 
layers obtained among the vertices of the graph in Figure 9.6 (i), starting from vertex s, are 
shown in Figure 9.6 (ii). The smallest number of such layers is 4. The graphs in Figure 9.6 
(iii)-(v) visually show step-by-step how to find the shortest path from vertex s to vertex v. It is 
clear that parent[v] = c, parent[parent[v]] = parent[c] = x, and parent[x] = s. So, (s, x, c, v) 
is the shortest path from s to v. Because vertex v occurs in the layer L,, the length of the 
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(iii) © 


(v) 


(vii) © 


Figure 9.6 Finding shortest paths by running breadth-first searches. 


above shortest path is 3. The graph in Figure 9.6 (vi) shows that (s, x, c, f) is a shortest path 
from vertex s to vertex f. 

Note that the shortest path may not be unique (in some cases may not even exist (see 
Remark 9.3)). This can be seen in Figure 9.6 (vi) and (viii). In fact, Figure 9.6 (vii) shows 
another breadth-first tree for the graph shown in Figure 9.6 (i), and accordingly Figure 9.6 
(viii) shows another shortest path from vertex s to vertex f, namely the path (s, x, d, f), which 
is different from that shown in Figure 9.6 (vi). a 


9.3.3 Testing Bipartiteness 


Breadth-first search can be used to test bipartiteness. The algorithm commences its 
exploration from any vertex and assigns alternating labels to the visited vertices during the 
search process. That is, give label 0 to the starting vertex, 1 to all its neighbors, 0 to those 
neighbors’ neighbors, and so on. Should a vertex, at any point during the process, possess 
neighboring vertices with identical labels (indicating they have been visited), it signifies 
that the graph is not bipartite. Conversely, if the search concludes without encountering 
such a scenario, it implies that the graph is bipartite. 

Workflow 9.2 uses graph coloring (see Theorem 4.11) and breadth-first search to test 
bipartiteness. We visually show the steps of this workflow in Figure 9.7. 
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(i) We test the bipartiteness of this graph. (ii) Assign neighbors of the source vertex 
Color source vertex, say gray. another color, say light blue. 


(iii) Assign all neighbors of the vertices (iv) Assign all neighbors of the vertices 
colored in (ii) the color gray. colored in (iii) the color light blue. 
O O 
@ @ 
O O 
O © 
O @ 
@ O 
(v) Repeat until all vertices are colored, ora (vi) Since no conflicting evidence was found, 
conflicting assignment occurs. the graph is bipartite. 


Figure 9.7 Illustrating the progress of breadth-first search on testing bipartiteness of a graph. 


Workflow 9.2 We test the bipartiteness of an input graph G by following five steps: 


(i) Assign a color (say red) to a random source vertex. 
(ii) Assign all the neighbors of the above vertex another color (say blue). 
(iii) Take one neighbor at a time to assign all the neighbor’s neighbors the color red. 
(iv) Continue in this manner till all the vertices have been assigned a color. 
(v) Ifat any stage, we find a neighbor which has been assigned the same color as that of the 
current vertex, stop the process. The graph cannot be colored using two colors. Thus 
the graph is not bipartite. 


Finding connected components is another application of breadth-first search that is not 
covered in this section. In Sections 9.4 and 9.5, we present depth-first search as an alterna- 
tive to breadth-first search. Since finding connected components is an application of both 
breadth- and depth-first searches and it can be described in both methods by the same 
scheme, finding connected components will be discussed together with the applications 
of depth-first search. 


9.4 Depth-First Search Algorithm 
9.4 Depth-First Search Algorithm 


In Sections 9.2 and 9.3, we have presented the breadth-first search method for exploring 
a graph. In this section, we present depth-first search as another method for exploring a 
graph. In Section 9.5, we will describe some of its applications. 

Recall that the breadth-first search involves a systematic exploration starting from a 
source vertex, extending in all feasible directions, and including vertices one layer at a time. 
In contrast, depth-first search distinguishes itself by visiting vertices in sequence until it 
encounters a “dead end,” after which it retraces its steps (backtracking); see Figure 9.8. 
Like breadth-first search, depth-first search is used for both directed and undirected 
graphs. 

In a depth-first search, we follow a path starting from a source vertex, say s, of a graph 
G = (V, E) until we get stuck, then we backtrack until we reach unexplored neighbor, being 
careful not to repeat a vertex. See Figure 9.9. 

Example 9.5 explains the steps outlined in the depth-first search method. A version of a 
depth-first search algorithm is presented in Algorithm 9.2. 


Algorithm 9.2: Depth-first search algorithm 


1: begin 
2: | parent = {s : None} 
3: | DFS-visit(V, Adj,s) 
4. | begin 
5: for v in Adj[s] do 
6: if v not in parent then 
7: parent[v] = s 
8: DFS-visit(V, Adj, v) 
9: end 
10: end 
11: | end 
12: | DFS(V, Adj) 
13: | begin 
14: parent = { } 
15: for sin V do 
16: if s not in parent then 
17: parent[s] = None 
18: DFS-visit(V, Adj, s) 
19: end 
20: end 
21: | end 


22: end 
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Figure 9.8 The operation of breadth-first search (a) versus that of depth-first search (b) on an 
undirected graph. 


In the context of the depth-first search, we make assump- 
tions about the arrangement of a vertex’s children based 
on their positions relative to the parent vertex. For ver- 
tices placed horizontally to the left or right of the parent, 
we consider their order from top to bottom as illustrated 
in the digraph shown to the right. In the case of ver- 
tices positioned vertically above or below the parent, we 
establish their order from left to right as illustrated in the 
graph shown to the right of Figure 9.8. This means that 
while exploring the neighborhood, we assume the chil- 
dren’s arrangement moves from top to bottom and left to 
right. 


Figure 9.9 The depth-first 
search scheme. 


Example 9.5 Show the progress of the depth-first search method on each of the following 
graphs starting from vertex s. 


(a) The undirected graph: (b) The directed graph: 


Solution 

(a) The progress of the depth-first search method on the graph given in item (a) is visualized 
in Figure 9.10 (i)-(iv). 

(b) The progress of the depth-first search method on the graph given in item (b) is visualized 
in Figure 9.10 (v)-(viii). a 


9.5 Applications of Depth-First Search 


Figure 9.10 Visualization of the progress of the depth-first search method on the graphs of 
Example 9.5. 


Theorem 9.2 gives the running time of Algorithm 9.2 on an input graph G = (V, E) given 
in its adjacency lists. This theorem concludes that the depth-first search algorithm runs in 
time linear in size of the adjacency list representation of G. 


Theorem 9.2 Algorithm 9.2 runs in O(V + E) time. 


Proof: Note that the function DFS-Visit in Algorithm 9.2 calls itself once for each vertex 
in V since each vertex is added to the resulting tree at most once. The for-loop within DFS - 
Visit function is executed a total of |E| times in the case of a directed graph, or 2|E| times 
for an undirected graph, as each edge is explored exactly once. The for-loop in the function 
DFS in Algorithm 9.2 adds O(V) time. Therefore, the total running time is O(V + E). This 
completes the proof. a 


9.5 Applications of Depth-First Search 
The depth-first search has some applications, and there are some O(V + E£)-time algorithms 


based on the depth-first search algorithm for the following problems: Computing a span- 
ning tree (forest) of a graph, detecting a cycle in a graph or reporting that no such cycle 
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exists, testing whether a graph is connected, and finding the connected component that 
contains a given vertex. Below we describe some of these applications. 


9.5.1 Computing Spanning Trees (Forests) 


For a given graph G = (V, E), the depth-first search algorithm directly produces a spanning 
tree rooted at the source vertex. If the given graph G is disconnected, then the depth-first 
search produces a spanning forest of G. The tree (forest) produced by performing a depth- 
first search is called a depth-first tree (forest). 

For a directed graph G, we define the following four edge types in terms of its depth-first 
tree (forest): 


e A tree edge: An edge of a depth-first tree (forest) of G. 

e Aback edge: A non-tree edge connecting a vertex to one of its ancestors (but not parents) 
in a depth-first tree (forest) of G. Note that a self-loop is a back edge. 

e A forward edge: A non-tree edge connecting a vertex to one of its descendants (but not 
children) in a depth-first tree (forest) of G. 

e A cross edge: A non-tree edge connecting some pair of vertices that have no 
ancestor—descendant relationship in a depth-first tree (forest) of G. 


For an undirected graph G, there are no forward edges because they become back edges 
when we move in the opposite direction, and there are no cross edges because every edge of 
an undirected graph G must connect an ancestor vertex to a descendant vertex. This leads 
us to the following remark. 


Remark 9.4 For an undirected graph G, there are tree edges and back edges only in terms 
of its depth-first tree (forest). No forward or cross edges. 


For the given digraph shown to the left of Figure 9.11, we compute a spanning forest by 
drawing the produced depth-first forest. The computed spanning forest consists of two trees 
with two different roots. 

As shown to the right of Figure 9.11, the tree edges are the thick ones and the vertices 
are enumerated in alphabetical order starting at a (the src vertex), where each vertex is also 
super-indexed either by (1) to mean that it is located on the first tree that has root a, or by 


src 


—p> Tree edge -= > Back edge 


—-— Frwdedge — ee > Cross edge 


Figure 9.11 A graph and its depth-first forest. 


9.5 Applications of Depth-First Search 


(2) to mean that it is located on the second tree that has root e. Note that (d, b) is a back 
edge, (a, d) is a forward edge, and (g, e) and (d, c) are cross edges. 


9.5.2 Detecting Cycles 


Detecting cycles is one of the applications of the depth-first search algorithm. The following 
remark gives evidence when the existence of a cycle in an input graph is affirmed. 


Theorem 9.3 A graph G has a cycle if and only if there exists a back edge in G after per- 
forming any depth-first search. 


Proof: To begin, let us establish that if a graph G = (V, E) contains a cycle, then it also 
includes a back edge. By performing a depth-first search on G, we can arrange the vertices 
in a descending order based on their appearance in the depth-first tree, starting from a 
source vertex. It is important to note that tree edges, forward edges, and cross edges all 
point forward in this ordering. Conversely, back edges point backward toward an earlier 
vertex within this sequence. Given that, according to our assumption, these vertices form a 
cycle, it logically follows that there must be at least one back edge. Otherwise, there would 
be no means to traverse from a later vertex in the ordering back to an earlier vertex, which 
directly contradicts the initial assumption of the existence of a cycle. 

Conversely, let us assume that (u, v) € E represents a back edge. Following the definition 
of back edges, there must exist a path P from vertex v to vertex u within the depth-first tree. 

Now, let us define a new set C as C = P + (u, v); this set C forms a cycle within graph G. 
The proof is complete. E 


Recall that a directed acyclic graph (DAG) is a digraph that has no directed cycles. 
Theorem 9.3 leads immediately us to the following corollary. 


Corollary 9.1 A digraph is a DAG if and only if it has no back edges after performing any 
depth-first search. 


Workflow 9.3 tells us how to detect and report a cycle in an input graph G. 


Workflow 9.3 We detect and report a cycle in an input graph G by following four steps: 


(i) Run a depth-first search. 
(ii) If at any stage, we find a back edge (u, v), stop the depth-first search process. A cycle 
is detected in the graph. 
(iii) Find the (unique) path P from v to u in the produced depth-first tree. 
(iv) Add the edge (u, v) to the path P to localize the detected cycle. 


We visually show the steps of Workflow 9.3 in Figure 9.12. After running a depth-first 
search on the graph shown in Figure 9.12 (i), the back edge (f, b) is found, as illustrated in 
Figure 9.12 (viii), and the cycle (b, c, g, f) is detected, as localized in Figure 9.12 (ix). 
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A depth-first search is applied. A back edge is found. A cycle is detected. 


Figure 9.12 Detecting a cycle by running a depth-first search. 


9.5.3 Finding Connected Components 


Depth- and breadth-first searches can be used to test if a graph is connected as follows: 
Starting from a random source vertex, if on termination of algorithm, all vertices are visited, 
then the graph is connected, otherwise it is disconnected. The following remark proves the 
correctness of this assertion. 


Remark 9.5 Depth- and breadth-first searches visit all vertices that are reachable from 
source vertex s. 


Proof: We prove this remark by induction. Firstly, both depth-first and breadth-first 
searches unquestionably cover all vertices at level 0, which comprises only the vertex s 
itself. This serves as our base case. Next, we consider the vertices located at the minimum 
distance i from vertex s and label them as “level i” vertices. If either the depth-first or 
breadth-first search successfully reaches all the vertices at level i, then it naturally follows 
that they will also reach all the vertices at level i+ 1, because every vertex at a distance of 
i+ 1 from s must be connected to some vertex at a distance of i from s. This constitutes the 
inductive step and completes the proof. a 


9.6 Topological Sort 


Depth- and breadth-first searches can also be used to find the connected components 
in an input graph (i.e., finding all vertices within each connected component). To find 
a connected component within a graph, we initiate the process by selecting an arbitrary 
source vertex and commence either a depth-first or breadth-first search from that vertex. 
All the vertices reachable from this chosen vertex constitute a single connected component. 
To uncover all the connected components within the graph, we systematically examine 
every vertex, discovering their connected components one by one through graph traversal. 
It is important to note, however, that there is no need to initiate a search from a vertex, say, 
v, if we have previously identified it as a part of a previously found connected component. 
Therefore, by maintaining a record of encountered vertices, we can streamline the process, 
ensuring that we only need to conduct one search for each connected component. 

The connected component, say R, which encompasses an initial vertex s, can be deter- 
mined by executing the code outlined in Algorithm 9.3. During a search starting from a 
specific vertex s, it is evident that we will never extend our reach to any vertices situated 
beyond the boundaries of the connected component when utilizing depth-first or breadth- 
first search methods. Consequently, by Remark 9.5, Algorithm 9.3 will find each connected 
component correctly. That is, upon termination, R will consist of vertices that are reachable 
from vertex s. 


Algorithm 9.3: Finding connected components 

1: R= {s} 

2: while (there is an edge (u, v) where u € R and v ¢ R) do 
3: | add v to R 

4: end 

5: return R 


There are other applications of depth-first search such as testing bipartiteness and topo- 
logical sorting. We decided to not include testing bipartiteness here because it was previ- 
ously described as an application of breadth-first search. Topological sorting will presented 
in Section 9.6 as an application of depth-first search. 


9.6 Topological Sort 


A topological ordering or sorting of an n-vertex directed graph G = (V, E) is an ordering of 
its vertices as v4, V2, -.. , U„ SO that for every edge (v;, v;) € Ewe have i <j. 

Less informally, a topological ordering arranges the vertices of a directed graph along a 
horizontal line so that all edges point from left to right; see Figure 9.13. Formally, we have 
the following definition. 


Definition 9.1 A topological ordering of a directed graph G = (V, E) is a total order “<” 
such that u < v for every edge (u, v) E E. 


Topological sorting is used to schedule tasks under precedence constraints. Assume that 
we have some tasks to execute, but certain tasks must be completed before others can begin. 


283 


284 


9 Elementary Combinatorial Algorithms 


See P @ P 


(b) 


Figure 9.13 A directed graph (a) and its topological ordering (b). 


e ox 
Rough wall g ° Wall board 


(a) (b) 


Figure 9.14 A house construction order, indicating which to execute first, is visualized as a 
digraph (a), while topological sorting reveals precedence constraints (b). 


This dependency between tasks can be visualized as a directed graph, and topological sort- 
ing comes to the rescue in scheduling tasks while adhering to these precedence constraints. 
Our objective is to ascertain whether there exists an order in which this set of tasks can be 
successfully executed, taking into account the specified constraints. This order, if it exists, 
essentially constitutes a topological sort of a digraph G. In this graph, the vertices rep- 
resent individual tasks, while the edges signify the precedence constraints governing the 
sequence in which these tasks must be undertaken. Edge (v, v,,,), for instance, means 
that task v; must be completed before task v,,, can be started; see Figure 9.14. Topolog- 
ical sort is often useful in selecting course prerequisites of a target program, for which 
the course v; must be taken before course v;,;. It is generally useful in scheduling jobs in 
their proper sequence, for which the output of job v; is needed to determine the input of 
job Vist 

Note that the graphs in Figures 9.13 and 9.14 are DAGs (recall that a DAG is a directed 
acyclic graph). We have the following theorem. 


Theorem 9.4 A digraph has a topological order if and only if it is a DAG. 


Proof: Let us begin by demonstrating that if a digraph G possesses a topological order, then 
G is a DAG. Contrarily, suppose that G does have a directed cycle. In such a scenario, it 
becomes evident that obtaining a topological ordering for G is impossible. This is because 


9.6 Topological Sort 


the rightmost vertex within the cycle would inevitably have an edge pointing toward the 
left, directly contradicting the existence of a topological order for G. 

Conversely, consider a DAG represented as G = (V, E). Our goal now is to establish that 
G indeed possesses a topological order. We begin by considering an arbitrary ordering 
designated as “<” for the vertices in G. In instances where v < u holds true for any edge 
(u, v) E€ E, it signifies that within G, there exists a directed path originating from vertex v 
and concluding at vertex u. Consequently, it also indicates the presence of a directed cycle 
encompassing the edge (u, v). Alternatively, if G lacks cycles (i.e., is acyclic), then u < v 
is applicable for any edge (u, v) € E. In simpler terms, any DAG inherently possesses a 
topological order. a 


The three-step procedure in the following workflow, followed by an example, will learn 
us how to compute a topological ordering of a DAG. 


Workflow 9.4 We compute a topological ordering of a DAG G = (V, E) by following three 
steps: 
(i) Find a vertex v € V with no incoming edges and order it first. 


(ii) Delete v from G. 
(iii) Recursively compute a topological ordering of G — {v} and append this order after v. 


Example 9.6 Compute a topological sort for each of the following DAGs. 


(a) The connected digraph: (b) The disconnected digraph: 


n n 


The progress of Workflow 9.4 steps to compute a topological sort for the DAG given in part 
(a) is visualized in Figure 9.15. A topological sort for the DAG given in part (b) is computed 
and shown in Figure 9.16. E 


It is not hard to find that the topological sort has finishing time of the reverse of depth- 
first search. We have shown in Theorem 9.2 that the depth-first search runs in O(V + E) 
time. Hence we can conclude the following corollary. 


Corollary 9.2 The topological sort runs in O(V + E) time. 


In conclusion, topological sorting stands as a fundamental concept in graph theory, 
providing a systematic arrangement of vertices in a DAG. Its practical applications span 
various fields, aiding in the optimization of scheduling, task sequencing, and dependency 
resolution. 
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Figure 9.15 Visualization of the progress of Workflow 9.4 steps to compute a topological ordering 
for the DAG in Example 9.6 (a). 
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Figure 9.16 The DAG in Example 9.6 (b) and its topological ordering (shown to the left). 


Exercises 


9.1 Choose the correct answer for each of the following multiple-choice questions/items. 


(a) Which one of the following statements is false? 
(i) The adjacency list is preferred when the graph is a cycle while the adjacency 
matrix is preferred when the graph is complete. 
(ii) The adjacency list is faster than the adjacency matrix for listing the vertices 
adjacent to a given vertex. 
(iii) The adjacency matrix is faster than the adjacency list for determining if there 
is an edge between two vertices. 
(iv) The adjacency matrix is preferred when the graph is sparse while the adja- 
cency list is preferred when the graph is dense. 

(b) If we use a breadth-first search for testing bipartiteness of an input graph, we stop 
the process when 

(i) all vertices are visited. 

(ii) all vertices are colored red or blue, or a conflicting color assignment occurs. 
(iii) a back edge is found. 
(iv) it exceeds 10 minutes. 

(c) If we run a depth-first search on a directed graph, and then remove all of the back 
edges found, the resulting graph is 

(i) a tree. (ii) cyclic. (iii) acyclic. (iv) bipartite. 

(d) Assume in some general connected graph G we run the breadth-first search and 
the depth-first search on G starting from the same vertex s and we find that the 
resulting trees are the same, then this necessarily means that G is 

(i) acycle. (iii) a complete graph. 
(ii) a tree. (iv) acomplete bipartite graph. 

(e) Suppose that G is a graph with n vertices and at most n log n edges, represented 
in its adjacency list representation. Then the run time of the depth-first search on 
Gis: 

(i) O(n). (ii) Odog n). (iii) O(n log n). (iv) O(n/ logn). 

(f) A person wants to visit some places. He starts from a vertex and then wants to 
visit every place connected to this vertex and so on. What algorithm he should 
use? 
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(i) Breadth-first search. (iii) Topological sorting. 
(ii) Depth-first search. (iv) Any of the above. 

(g) Consider the undirected graph G = (V,E), where V = {m,n,0,p,q,r} and E = 
{(m, n), (n, q), (q, m), (n, 0), (0, p), (p, q), (m, r)}. If we run the breadth-first search 
on G starting at any vertex, which one of the following is a possible order for 
visiting the vertices? 

(i) m,n,0,p,q,r. (ii) n,q,m,p,o,r. (iii) r,m,q,p,o,n. (iv) q,m,n, p,r,0. 

(h) Suppose that G is a graph with n vertices and at most n? logn edges, represented 
in its adjacency list representation. Then the run time of the breadth-first search 


on G is: 
(i) O(n). (ii) O(log n). (iii) O(n log n). (iv) O(n? logn). 
(i) If we use a depth-first search for detecting cycles in an input graph, we stop the 
process when 


(i) all vertices are visited. 

(ii) all vertices are colored red or blue, or a conflicting color assignment occurs. 
(iii) a back edge is found. 
(iv) it exceeds 10 minutes. 

(j) Three students were given a DAG, G = (V,E), where V = {2, 3, 5,7, 8,9, 10, 11} 
and E = {(5,11), (7,8), (7,11), (3,8), (3, 10), (8, 9), (11, 2)(11, 9), (11, 10)}. Each 
one of these students run the topological sorting on G twice and obtained a result 
of the following. Which student made a mistake? 

(i) The first student who got the sorts: 5, 7, 3, 11, 8, 2, 9, 10 and 3, 5, 7, 8, 11, 2, 


9, 10. 

(ii) The second student who got the sorts: 3, 5, 7, 8, 11, 2, 9, 10 and 5, 7, 3, 8, 11, 
10, 9, 2. 

(iii) The third student who got the sorts: 7, 5, 11, 3, 10, 8, 9, 2 and 5, 3, 11, 2, 7, 8, 
9, 10. 


(iv) No student made a mistake. 


9.2 Read each of the following statements and decide whether it is true or false. 


(a) For undirected graphs, in both the adjacency list and the adjacency matrix, there 
are two representations of each edge in the graph. 

(b) Breadth- and depth-first searches can both be used for testing the connectedness 
of a graph. 

(c) A graph G is acyclic if and only if there does not exist a back edge in G after per- 
forming any depth-first search. 

(d) We can find a topological sorting for any digraph, even if it contains a dicycle. 

(e) If a topological sort exists for the vertices in a graph, then a depth-first search on 
the graph will produce no back edges. 

(f) For directed graphs, in both the adjacency list and the adjacency matrix, there are 
two representations of each edge in the graph. 

(g) Breadth- and depth-first searches can both be used for finding connected compo- 
nents. 

(h) Given two vertices of a graph G, the shortest path in G between them is not always 
unique. 


9.3 


9.4 


9.5 
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(i) For an undirected graph G, there are tree edges and forward edges only in terms 
of its depth-first tree (forest). No back or cross edges. 

(j) A digraph G has a topological order if and only if there exists a back edge in G 
after performing any depth-first search. 


Represent the graph in Example 9.1 (c) using: 


(a) Adjacency list representation. (b) Adjacency matrix representation. 


Consider the following two graphs; one of them is undirected and the other is directed. 
Let v, be the source vertex in each. 


© D A 
O% OW 
© © © 


(a) We run a breadth-first search on the undirected graph and obtained the breadth- 
first tree in Figure 9.5. Use this to find the shortest path from vertex v, to vertex 
vg. Is this shortest path unique? 

(b) Use breadth-first search to test the undirected graph for bipartiteness. 

(c) Run a depth-first search on the directed graph to compute a depth-first tree. Clas- 
sify the edges as tree edges, back edges, forward edges, and cross edges. 

(d) Does the directed graph have a directed cycle? Why or why not? Link your answer 
to the depth-first search completed in item (c). 

(e) Compute a topological ordering for the directed graph. 


3 


Consider the following the directed graph, and let s be its source vertex. 


A Q 


(a) Run a breadth-first search on the graph to obtain the breadth-first tree. 

(b) Use breadth-first search to test the undirected version of the graph for bipartite- 
ness. 

(c) Run a depth-first search on the graph to compute a depth-first tree. 

(d) Does the graph have a directed cycle? Why or why not? Link your answer to the 
depth-first search completed in item (c). 

(e) Compute a topological ordering for the graph. 
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9.6 Answer items (a) through (e) in Exercise 9.5 for the following graph, which has s as 
its source vertex. 


9.7 Prove or disapprove: A depth-first search of a directed graph always produces the same 
number of tree edges (i.e., independent of the order in which the vertices are provided 
and independent of the order of the adjacency lists). 


Notes and Sources 


The history of combinatorial algorithms, particularly breadth-first search and depth-first 
search, can be traced back to the early 19th century. These graph algorithms were first 
described by Charles Pierre Trémaux in 1857 for solving mazes and puzzles. The earliest 
reference to depth-first search is referred to Gaston Tarry in 1883, while the first explicit 
description of the breadth-first search was published in 1957 by Edward Moore (refer to 
Moore (1959)). Over the years, combinatorial algorithms have become essential tools in 
computer science and graph theory, used for traversing and analyzing the structure of 
graphs, trees, and networks. 

This chapter introduced these fundamental algorithms designed for graph exploration 
and search. To prepare adequately for these algorithms, we first learned the techniques for 
graph representation. The primary emphasis of the chapter was on the two key searching 
algorithms: Breadth-first search and depth-first search. We also described the applications 
of these algorithms in computing spanning trees, finding shortest paths, testing bipartite- 
ness, detecting cycles, and identifying connected components. Additionally, we presented 
the topological sort for ordering DAGs. 

As we conclude this chapter, it is worth noting that the cited references and others, 
such as Erickson (2019); Maurer and Ralston (2004); Hougardy and Vygen (2016); Even 
and Even (2011); Sedgewick (2003); Thulasiraman and Swamy (2011); Lawler (2001); 
Kadek and Panovics (2014); Ajwani et al. (2006); Klöckner et al. (2004); Goldfarb et al. 
(1991); Manber (1990); Sikléssy et al. (1973); Allender et al. (2022); Nishihara and 
Minamide (2004); Berghammer and Hoffmann (2001); Tarjan (1972); Hagerup (1990), 
and Aggarwal et al. (1990), also serve as valuable sources of information pertaining to 
the subject matter covered in this chapter. The code that created Figure 9.8 is due to 
StackExchange (2016). The code that created Figure 9.9 is due to StackExchange (2013). 
We used and modified a code due to StackExchange (2019) to create the adjacency 
matrix representations in Section 9.1. We used and modified a code due to Stack- 
Exchange (2020) to create the adjacency list representations in Section 9.1. We used 
some parts of a code due to StackExchange (2017) to draw topological sort diagrams in 
Section 9.6. 
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Optimization is a term used to describe any step-by-step procedure created to find the best 
solution to a problem with more than one solution. Ever since its inception in the 1940s, 
initially in the context of military planning, linear optimization (originally and still known 
as linear programming (LP for short)) has found extensive application across various indus- 
tries and disciplines. 

Let f : [a,b] > R be a nonlinear continuous 
function. In Calculus, if we want to minimize/- 
maximize the function f(x), we must take the 
derivative, and then find the critical points. We 
also check the endpoints, if there are any. We can 
justify our maxima or minima either by the first 
derivative test, or the second derivative test. In 
the graph shown to the right, a and b are end- 
points of the function f(x), and c and d are their 
critical numbers (f’(x) = 0 when x = c, d). The 
function f has maximum values at x = a,d, and 
has minimum values at x = c, b. 


y 
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Now, instead of optimizing a nonlinear 
function on [a, b], consider a linear function 
on [a,b]. In this case, we need to check 
only the endpoints as in the figure shown 
to the right. Linear optimization studies 
generalizations of this easy (linear) case 
to higher dimensions. More specifically, 
instead of optimizing a linear function of 
only one variable, say cx, on the closed 
interval [a,b], we optimize a linear func- 
tion of a finite number of variables, say 
clx = CX, +x, +---+C,X,, on polytopes, 
which are generalizations of polygons from 
R? to R”, where the set R” consists of all 
n-tuples of real numbers, R. This study is 
“easy” to understand because of linearity, but 
it is “difficult” to carry out because of high 
dimensionality. 


A linear function 


In this chapter, we introduce linear programming, the graphical method, and study the 
LP duality and geometry. We also study the most common linear programming algorithm, 
the simplex method. Over and above that, we study an interior-point method as one of the 
non-simplex methods. The references Bertsimas and Tsitsiklis (1997) and Nemhauser and 
Wolsey (1988), for example, is a good source for information relative to this topic. 


10.1 Linear Programming Formulation and Examples 


In this section, we will see that applications of LP touch a vast range of real-world areas. 
First, we present the general form of an LP problem. 


10.1.1 General Form Linear Programs 


An LP problem is the problem of minimizing a linear cost function subject to linear equality 
and inequality constraints. We have the following example. 


Example 10.1 The following is an LP problem. 


minimize 4x,—x, +3x, 


subjectto x, +x, +X,<7, 
2X,-X; = 6, 

x, +xX,> 4, 

x, 2 0, 


X3 <0. 


10.1 Linear Programming Formulation and Examples 


Here, x,,X,,X,, and x, are the decision variables whose values are to be chosen to min- 
imize the linear cost function 4x, — x, + 3x, subject to linear equality and inequality con- 
straints. E 


Generally speaking, assume that we are given a cost vector € = (C), C3, ... C) and 
we minimize a linear cost function c'x =), cx; over all nth-dimensional vectors 
X = (X,,X),...,X,)' subject to linear equality and inequality constraints. Then we are 
interested in a problem of the form: 

min c'x 
s.t. alx >b, i=1,2,...,m,, 
alx< b, j=1,2,...,m, 


a 


10.1 
ajx = b,, k=1,2,...,ms, (10.1) 


X, 20, p=1,2,...,m,, 
Xq <0, q = 1,2,..., M5. 


Problem (10.1) is said to be the general form LP. We have the following definition. 


Definition 10.1 Consider the minimization problem (10.1). Then: 


(a) The variables x,,x,,...,x,, are called decision variables; 

(b) A vector x satisfying all of the constraints is called a feasible solution; 

(c) The set of all feasible solutions is called the feasible set or feasible region; 

(d) Ifx, > 0 or x; < 0, then x; is called a restricted variable; otherwise, it is called a free or 
unrestricted variable (urs); 

(e) The function c'x is called the objective function or cost function; 

(£) A feasible solution x* that minimizes the objective function (i.e., c'x* < cx for any 
feasible solution x) is called an optimal solution; 

(g) The value of c'x*, corresponding to an optimal solution x*, is called the optimal cost 
or optimal value; 

(h) If the optimal cost is —co, we say that the minimization problem is unbounded. 


Example 10.2 Consider the following nonlinear minimization problem. 


min 2x, + |x] 

S.t. 5x + 7x, <3, 
|x| +x, <4, 
X,,X, urs. 


Using the fact that |t| = max {t, —t } for t € R, this problem can be expressed as 


min 2x,+y 

s.t. 5x, + 7x, < 3, 
X,+xX, <4, 
—X, +x, <4, 
y=% 20, 
ytx, 20, 
X,,X, urs, 


which is an LP problem. a 
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Note that there is no need to study maximization problems separately because maximiz- 
ing c'x subject to some constraints is equivalent to minimizing (—c)'x subject to the same 
constraints (why?). 


10.1.2 Examples of Linear Programming Problems 


This part presents some examples of LP problems and allows the reader gain to some famil- 
iarity with the art of constructing mathematical optimization models. 

The procedure given in the following workflow, followed by some examples, will teach 
us how to formulate linear optimization models. 


Workflow 10.1 There are three steps involved in the formation of an LP problem: 


(i) Identify the decision variables of interest to the decision-maker and express them as 
Hi ye Kapsa 
(ii) Ascertain the objective function in terms of the decision variables. This would be a cost 
in case of minimization problem or a profit in case of maximization problem. 
(iii) Ascertain the constraints representing the maximum availability or minimum com- 
mitment. 


Example 10.3 (Maximizing profit in product manufacturing) 

A company is involved in the production of two items, denoted as P} and P,. The manu- 
facturing process for each unit of product P} necessitates 2 kg of raw material and 4 labor 
hours for processing, while each unit of product P, requires 5 kg of raw material and 3 labor 
hours of the same type. On a weekly basis, the company has access to 45 kg of raw material 
and 55 labor hours. For the financial aspect, the company gains a profit of JD 25 for every 
unit of product P} sold and JD 35 for every unit of product P, sold. Formulate this problem 
as an LP problem that maximizes the total profit. 


Solution 
The given data can be summarized in the following table. 


Product Row material Labour hours Profit 
P 2kg 4h JD 25 
P, 5kg 3h JD 35 
Restrictions 45kg 55h 


e The first step is to identify the decision variables. Let x, denote the number of units that 
should be produced from product P, per week, i = 1, 2. 

e The second step is to determine the objective function. The objective is to maximize the 
total profit. So, our objective function is z = 25x, + 35x). 

e The third step is to formulate the constraints. In this example, the constraints are: 


e Arestriction on the row material. This can be formulated as 2x, + 5x, < 45. 
e A restriction on the labor hours. This can be formulated as 4x, + 3x, < 55. 
e Nonnegativity constraints. This can be formulated as x, > 0 and x, > 0. 


10.1 Linear Programming Formulation and Examples 


As a result, this problem can be formulated as the following LP model. 


max 25x, + 35x, 

s.t. 2x, + 5x, < 45, 
4x, + 3x, < 55, 
XxX% > 0. 


Example 10.4 (Maximizing profit in corn chip production) 

A corn chip company operates with two distinct departments, each responsible for pro- 
ducing two types of corn chips: “extra larges” and “really smalls.” The company earns a 
profit of 225 per kilobag of extra larges and 175 per kilobag of really smalls (where a kilo- 
bag contains 1000 bags). Each department adheres to specific production regulations per 
day. The company’s primary objective is to maximize its profit while complying with these 
regulations. 


(a) Identify the decision variables. 
(b) Write the objective function z in terms of the decision variables. 
(c) Write inequalities expressing the following constraints: 


(i) The production of extra larges should not exceed 20 kilobags per day, and the pro- 

duction of really smalls should not exceed 30 kilobags per day. 

(ii) No more than a total of 45 kilobags can be produced each day. 

(iii) The number of extra larges produced daily must be at least 2/3 of the number of 
really smalls produced. 

(iv) The company must utilize more than 250 hours of labor each day to satisfy union 
requirements. Making one kilobag of extra larges consumes 10 hours and making 
one kilobag of really smalls consumes 15 hours. 


Solution 
(a) The decision variables are: 


x: The number of extra large corn chips produced per day; 


y: The number of really small corn chips produced per day. 


(b) z = 225x + 175y. 


(c) (i) x < 20, y < 30. (iii) x > sy. 
(ii) x+y < 45. (iv) 10x + 15y > 250. a 


Example 10.5 (Minimizing nutritional costs) 

Two different food items, denoted as F; and F,, contain vitamins A and B. Food F; provides 
2 units of vitamin A and 5 units of vitamin B per unit, while food F, offers 4 units of vitamin 
Aand 2 units of vitamin B per unit. The cost of one unit of food F; is JD 10, and for food F,, it 
is JD 12.50. The objective is to meet or exceed the minimum daily nutritional requirements 
for vitamins A and B, which are 40 and 50 units, respectively, at the lowest possible cost. 
Formulate this problem as an LP problem. 


299 


300 | 10 Linear Programming 


Solution 
The given data can be summarized in the following table. 


Food/vitamin A B Cost 

F, 2 units 5 units JD 10 
F, 4 units 2 units JD 12.5 
Restrictions 40 units 50 units 


Let x; be the number of units that should be daily produced from food F, for a person, 
i = 1,2. This problem can be formulated as the following LP model. 


min 10x, + 12.5x, 

s.t. 2x, + 4x, > 40, 
5x; + 2x, > 50, 
XxX% => 0. 


Example 10.6 (Maximizing advertising audience) 

A marketing manager has an annual advertising budget of JD 25,000, which he intends to 
allocate to two advertising media, A and B. Media A, a monthly magazine, costs JD 1000 
per message, and media B costs JD 1500 per message. The following conditions apply: For 
media A, not more than one insertion is desired in the issue. For media B, at least five 
messages should be placed. The expected effective audience for one message in media A 
is 40,000 people, while for media B, it is 50,000 people. Formulate this problem as an LP 
problem to maximize the total audience reached through advertising while staying within 
the budget constraints. 


Solution 
The given data can be summarized in the following table. 


Media Media A Media B Restrictions 
Audience 40,000 people 50,000 people 

One message cost JD 1000 JD 1500 JD 25,000 
Number of messages At most 1 At least 5 


Let x, and x, be the number of messages that should appear in media A and B, respec- 
tively. This problem can be formulated as the following LP model. 


max 40, 000x, + 50, 000x, 
s.t. 1000x; + 1500x, < 25, 000, 
x, <1, 
xX, 2 5, 
XX = 0. a 


10.1 Linear Programming Formulation and Examples 


Example 10.7 (Minimizing cost in sheep nutrition) 

A farmer is actively involved in breeding sheep, and the sheep’s diet primarily consists of 
various products grown on the farm. To ensure that the sheep receive the required nutrient 
constituents, the farmer must consider purchasing additional products, which we will refer 
to as Product A and Product B. The essential nutrient constituents (vitamins and protein) 
contained in each of these products are detailed in the table below: 


Nutrient Nutrient in product Nutrient in product Minimum requirement 
Constituents A B of nutrient constituents 
X 36 6 108 

Y 3 12 36 

Z 20 10 100 


Product A is priced at JD 20 per unit, while Product B is priced at JD 40 per unit. Formulate 
an LP problem that can minimize the total cost and satisfy the requirements. 


Solution 
Let x, and x, be the number of units that must be purchased from products A and B, respec- 
tively. This problem can be formulated as the following LP model. 


min 20x, + 40x, 
s.t. 36x, + 6x, > 108, 
3x, + 12x, > 36, 
20x, + 10x, = 100, 
X1,X, 2 0. | 


Example 10.8 (Nurse scheduling at a university hospital) 

A university hospital is seeking your assistance in scheduling nurses for their intensive 
care unit. In this scenario, it is assumed that the same daily schedule repeats, and the 
nurse requirements remain constant. Each workday is divided into four shifts: 12ZAM-6AM, 
6AM-12PM, 12PM-6PM, and 6PM-12AM. Every day, each nurse is assigned to work two 
of these shifts. Nurses working two consecutive shifts are compensated at a rate of $20 per 
hour, while those working a “split schedule” (e.g., 12AM-6AM and 12PM-6PM) receive $25 
per hour. (It is important to note that the shifts 6PM-12AM and 12AM-6AM are considered 
consecutive.) The following table indicates the daily nurse requirements for each shift: 


Shift Number required 
12AM-6AM 5 
6AM-12PM 12 
12PM-6PM 7 
6PM-12AM 10 


Formulate an LP that can assist this hospital in determining the optimal nurse scheduling 
to meet daily requirements and minimize the total nurse compensation cost. 
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Solution 
The decision variables are: 


xı: The number of nurses that work from 12AM to 12PM; 

x,: The number of nurses that work from 6AM to 6PM; 

x: The number of nurses that work from 12PM to 12AM; 

x,: The number of nurses that work from 6PM to 6AM; 

x;: The number of nurses that work from 12AM to 6AM and 12PM to 6PM; 

xs: The number of nurses that work from 6AM to 12PM and 6PM to 12AM. 
Minimizing the total cost, we obtain the following LP problem. 


min 20x,+20x,+20Xx;+20Xx,+ 25x5+ 25X6 


s.t. x+ Xy+ Xs > 5, 
Xx+ X+ X 2 12, 
X+ X+ Xs 27, 
X+ Xy+ X = 10, 
xX; X, X3, X4, Xs, x, > 0 ey 


10.2 The Graphical Method 


In this section, we discuss the graphical method for linear optimization problems of two 
variables. We will also visually demonstrate different LP cases which may result in different 
types of solutions. We start by presenting the following workflow of six steps to find the 
extremum (maximum or minimum) solution graphically. 


Workflow 10.2 The following steps involved in solving two-dimensional LP problems 
graphically: 


(i) Graph constraint equations on a rectangular coordinate plane. 
(ii) Determine the valid side of each constraint equation. 
(iii) Isolate and identify the feasible region. 
(iv) Determine the direction of improvement. 
(v) Locate the extreme corner. 
(vi) Find the optimum solution and the corresponding optimal value. 


As a direct application of the above steps, we have the following examples. 


Example 10.9 Use the graphical method to solve the following LP problem. 


min z= 2x + 5y 
s.t. 3x + 2y < 6, 
=x + 2y <4, 
x+y 21, 
x,y 20. 


10.2 The Graphical Method 


Solution 

Following the steps in Workflow 10.2, we obtain the graphical solution visualized in 
Figure 10.1. Note that the given objective function z = 2x + 5y is perpendicular to the 
vector c = (2,5)! for any given scalar z. For simplicity, we represent this using the vector 
c in Figure 10.1. Furthermore, decreasing z corresponds to moving the line z = 2x + 5y 
in the direction of —c. Therefore, to minimize z, we move the line 2x + 5y = z as much 
as possible in the direction of —c, as long as we do not leave the feasible region. From 
Figure 10.1, we find that the unique optimal solution is x = (1, 0)' and the optimal value is 
Z=2X14+5x0=2. E 


For a system of linear equations Ax = b, we have three possibilities: The system has a 
unique solution, it has infinitely many solutions, or it is inconsistent. For an LP, we have 
the corresponding three possibilities, but we have one more possibility in addition. An LP 
problem may have: 


e A unique/finite optimal solution; 

e No bounded solution (so the LP is unbounded); 

e No feasible solution (so the LP is infeasible); 

e Alternative (multiple or infinite number of) optimal solutions. 

In the context of graphical method, it is easy to visualize these four different cases, as will 
be evident from the following examples. 


Direction of c 


2x + 5y = 12.25 ~ 


Direction of —c 


Figure 10.1 Graphical solution of the LP problem in Example 10.9 
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Example 10.10 Use the graphical method to solve the following LP problems. 


(a) max z = 13x,+23x, (b) max z=x,+ xX 
s.t. xı + 3x, < 96, s.t. X,+3x, < 96, 
x+ x, < 40, x+ x, < 40, 
7X, + 4x, < 238, 7X, +4x, < 238, 
xX; 20. xX, X% 20. 
Solution 


(a) The graphical representation of the given LP problem is shown in Figure 10.2, with the 
feasible region shaded in cyan. From the graph, we find that the maximum value for z 
is 800 at x = (12, 28)". So, this LP problem has a unique optimal solution. 

(b) The graphical representation of the given LP problem is shown in Figure 10.3, with the 
feasible region shaded in cyan. Note that the z-line hits the entire line segment between 
the points (12, 28) and (26, 14). From the graph, we find that the maximum value for z is 
40, and that every point in the line segment between (12, 28) and (26, 14) is an optimal 
solution. So, this LP problem has alternative optimal solutions. a 


Example 10.11 Use the graphical method to solve the following LP problems. 


(a) min z= 3x,+x, (b) max z= 3x,+x, 
s.t. 5x; +X, = 42, s.t. 5X, +X = 42, 
2x, +X, => 30, 2X, +X, = 30, 
X, X, 20. Xx, X,2 0. 


13x; + 23x2 = 800 ~| 


Figure 10.2 Graphical solution of the optimization problem in Example 10.10 (a). 


10.2 The Graphical Method 


xit x = 20. 
N 


‘N 


xı+ x = 10. 20 


xı + 3x2 < 96 


xı +x. < 40 
—20 


Figure 10.3 Graphical solution of the optimization problem in Example 10.10 (b). 


Solution 

(a) In Figure 10.4, we have provided a graphical representation of the LP problem at hand. 
The feasible region is distinctly shaded in a cyan color for clarity. Upon inspecting the 
graph, we can readily deduce that the lowest attainable value for the objective function 
z occurs at 34. This minimal value of z is achieved when the decision variables are set 
tox = (4, 22)". 

(b) We provide a visual depiction of the given LP problem in Figure 10.5. Within this 
graph, the feasible region is distinctly highlighted in cyan. One can observe that the 
z-line, which represents the objective functions values, can be continuously extended 
toward the upper-right corner of the feasible region without any bound or limit. This 
observation implies that there is no finite or optimal value of z that can be achieved 
within the problem’s constraints. Consequently, we can conclude that this LP problem 
is unbounded, emphasizing the open-ended nature of this particular problem. a 


Example 10.12 Use the graphical method to solve the following LP problems. 


(a) max z= 13x,+23x, (b) max z= 13x,+23x, 
s.t. X,+ 3x, < 96, s.t. x+ 3x, > 96, 
x+ xX, > 30, x+ xX, < 30, 
7X,+ 4x, < 238, 7X,+ 4x, > 238, 
X, X20. Xx, X20. 
Solution 


(a) The graphical representation of the given LP problem is shown in Figure 10.6, with the 
feasible region shaded in cyan. From the graph, we find that the maximum value for z 
is 839.52 at x = (19.41, 25.53)". So, this LP problem has a unique optimal solution. 
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Figure 10.5 Graphical solution of the optimization problem in Example 10.11 (b). 


(b) The graphical representation of the given LP problem is shown in Figure 10.7. Note 
that there are no feasible solutions, that is, there are no points satisfying all constraints. 
Therefore, the feasible region is empty, and the LP problem is infeasible. a 


10.2 The Graphical Method 


13x, + 23x2 = 839.52 


xa 3x3:S: 96 


x1 + 3x2 > 96 


Figure 10.7 Graphical solution of the optimization problem in Example 10.12 (b). 


Example 10.13 For the LP problems given in Examples 10.10-10.12, indicate which case 
the LP belongs to (i.e., if the LP has a unique optimal solution, has many optimal solu- 
tions, is unbounded, or is infeasible), and which type the feasible region is found (i.e., if the 
feasible region is bounded, unbounded, or empty). 


Solution 
The answer is given in Table 10.1. a 
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Table 10.1 The answer of Example 10.13 


LP problem 


The LP in Example 10.10 (a) 
The LP in Example 10.10 (b) 
The LP in Example 10.11 (a) 
The LP in Example 10.11 (b) 
The LP in Example 10.12 (a) 
The LP in Example 10.12 (b) 


LP case/type 


Unique optimal solution 
Alternative optimal solutions 
Unique optimal solution 
Unbounded LP 

Unique optimal solution 
Infeasible LP 


Feasible region type 


Bounded 
Bounded 
Unbounded 
Unbounded 
Bounded 


Empty 


Example 10.14 Consider the following LP problem. 


max y 

st. -x+ y<l, 
3x +2y < 12, 
2x + 3y > 12, 
xX, y>0. 


(a) Sketch the feasible region of this LP and solve it using the graphical method. 
(b) Generally speaking, if (some of) the variables are restricted to be integer-valued, then 
the underlying optimization problem is called an integer (a mixed-integer) program. 


3x + 2y < 12 


Figure 10.8 Graphical solution of the optimization problem in Example 10.14. 


10.3 Standard Form Linear Programs 


In this example, assume that x and y are restricted to be integer-valued. Sketch its fea- 
sible region and solve it graphically. 


Solution 
(a) The graphical representation of the LP is shown in Figure 10.8, with the feasible region 
shaded in cyan. We find that the optimal solution is 3 at x = (2, 3)". 


(b) Introducing the condition x,y € Z changes the feasible region, which is now indicated 
by the blue bullet shown in Figure 10.8. The optimal solution remains the same. a 


10.3 Standard Form Linear Programs 


Recall that the general form LP is: 


min c'x 
s.t. alx > b;, i= 1,2, ..., Mı, 
T ; 
a,x < bj, j=1,2,...,m), 
(10.2) 
ajx = by, k =1,2,...,M, 
xX, 2 0, Pp = 1,2, ..., M4, 
Xq <0, q = 1,2,..., M5 


where c, x € R”. 
Recall also that there is no need to study maximization problems separately because maxi- 


mizing c'x subject to some constraints is equivalent to minimizing —c'x subject to the same 
constraints. In addition, because: 


e alx = b; is equivalent to alx < b, and a/x > b;; 

e alx < b; can be written as (—a,)'x > —b;; 

e x; > 0 and x, < 0 are special cases of u'x > 0 and (—u)'x > 0, respectively, where u is a 
unit vector in R”, 


Problem (10.2) can be expressed exclusively in terms of inequality constraints of the form 
alx > b;. As a result, Problem (10.2) can be formulated in vector form as 


min c'x 
(10.3) 
s.t. alx >b, i=1,2,...,m, 
or, more compactly, in matrix form as 
min c'x 
(10.4) 
s.t. Ax > b, 
where A e R”*" is the matrix whose rows are the row vectors aj,al,...,al, and 


b = (b,,b,,..., bay We have the following example. 
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Example 10.15 The LP problem in Example 10.1 can be written as 


min 2x, — x,+4x, 


st. -x-2 X, —X, 2-2, 
3X, — Xs 2 5, 

—3X, + x3 2-5, 

X3+X,2 3, 

x, 2 0, 

— X = 0. 


This can be also written in the matrix form (10.4) with 


-1 -l 0 -1 -2 
2 0 3 -1 0 5 
c= = , A= D ee , and b= => 
4 0 0 1 3 
0 1 0 0 0 
0 0 -1 0 0 m 
An LP problem of the form 
min c'x 
s.t. Ax=b, (10.5) 
x>0 


is said to be the standard form LP problem. 
We can convert an LP problem to the standard form by eliminating of free variables and 
eliminating of inequality constraints as detailed in the following workflow. 


Workflow 10.3 We can convert an LP problem to the standard form by following three 
steps: 


(i) Elimination of free variables: We replace each unrestricted variable x, with xt -X , 
where r e o >0. 
(ii) Elimination of “<” constraints: We replace Èa a,x; < b; with Da ajx; +s; = b;, 
where s; > 0 is called a slack variable (see also Definition 10.11). 
(iii) Elimination of “>” constraints: We replace 2a a,x, 2 b; with LA ax, — e; = b;, 
where e; > 0 is called an excess variable (see also Definition 10.11). 


Example 10.16 The LP problem: 


min 3x, + 7x, isequivalentto min 3x, + 7x} — 7x; 

st. x,+x, 23, standardform st. x, +x} —x;—x,=3, 
5x, + 3x,=19, LP problem: 5x, + 3x} — 3x7 =19, 
x >0, (letting x, = s3) be Alle, Se >0. 


For instance, given the feasible solution (x,,x,) = (2, 3) to the original problem, we obtain 
the feasible solution (1.x X3: X3) = (2, 3,0, 2) to the standard form problem. In Exercise 
10.12, we seek the point (x,,x,) for the original problem given the feasible solution 


(X,, Xs X;»X3) = (4,0, 1/3, 2/3) to the standard form problem. E 
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The graphical method for linear optimization problems indicates that an optimal solution 
to an LP lies at a “corner” of a polyhedron. A vertex, an extreme point, and a basic feasible 
solution all describe corners of a polyhedron, with the first two being geometric definitions. 


10.4.1 Extreme Points, Vertices, and Basic Feasible Solutions 


In this part, we define a vertex, an extreme point, and a basic feasible solution of a given 
nonempty polyhedron. 


Definition 10.2 Let P be a nonempty polyhedron. A vector x € P is called a vertex of P if 
there is some c such that c'x < c'y for all y € P different from x. 


From Definition 10.2, we observe that x is a vertex of a polyhedron P if it is the optimal 
solution of some linear program with P as the feasible region. In Figure 10.9, we show two 
polyhedra. In each polyhedron P, the hyperplane {y : c'y = c'v} on the right-hand side 
touches P at a single point, v, so the point v is a vertex. In contrast, the point w is not a 
vertex since there is no hyperplane intersecting solely at w within P. 


Definition 10.3 Let P be a nonempty polyhedron. A vector x € P is called an extreme 
point of P if there are no y, z € P and a scalar å € (0, 1) such that x = Ay + (1 — A)z. 


In Figure 10.10, we show three polyhedra. In each polyhedron, the vectors v,’s are extreme 
points, and the vector w is not an extreme point because w is a convex combination of v, 
and v,. 

Definitions 10.2 and 10.3 are geometric, and hence intuitive. We need an equivalent def- 
inition that is algebraic, so that we can do computations. Before this, we need an interme- 
diate concept for connecting geometry and algebra. 


w 
v 
Figure 10.9 Vertices (v’s) versus nonvertices (w’s). 
X2 Xi V4 
V2 vI X2 
Ww 
w v2 vi 
v2 
o val X1 o V3 X3 V3 


Figure 10.10 Extreme points (v;'s) versus nonextreme points (w’s). 
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Definition 10.4 Ifa vertex x* satisfies an inequality a'x > b (or a'x < b) as an equality, 
that is, a'x* = b, then we say that this inequality is active or binding at x*. 


If P c R” isa polyhedron defined by linear equality and inequality constraints, then x* € 
R” may or may not be feasible with respect the constraints. Now, ifx* € R” is feasible (i.e., 
x* € P; satisfying all the constraints), then from Definition 10.4 all the equality constraints 
are active at x*. 

We have the following example to more illustrate Definition 10.4. 


Example 10.17 The polyhedron shown in the middle of Figure 10.10 is expressed as 
P = {(%1:X2X3)! | xX +% +X = 155,43 > 0}. (10.6) 


There are three constraints that are binding at each of the points v,,v,, and v,. Namely the 
constraints x, + Xx, + x, = 1, x, = Oandx, = 0 are active at v,, the constraints x; + Xx, +x; = 1, 
x, = 0 and x, = 0 are active at v,, and the constraints x, +x, +x, = 1, x, = 0 and x, = 0 
are active at v,. Also, at the point w, there are two constraints that are binding, which are 
x +x% +X% = 1 and x = 0. a 


If there are n constraints that are binding at a vector x* € R”, then x* satisfies a system 
of n linear equations in n unknowns. In view of Theorem 3.13, this system has a unique 
solution if and only if these n equations are linearly independent. 

Now, we are ready to introduce the algebraic definition of a corner point. 


Definition 10.5 Let P be a polyhedron defined by linear equality and inequality con- 
straints, and x* € R”. We say that the vector 


(a) x* is a basic solution if the following two statements hold: 
(i) All equality constraints are active. 
(ii) Out of the constraints that are active at x*, there are n of them that are linearly 
independent. 
(b) x* is a basic feasible solution if it is a basic solution, and satisfies all of the constraints 
(i.e.,x* € P). 


The following two examples illustrate Definition 10.5. 


Example 10.18 

In the polyhedron depicted in the middle of Figure 10.10, as repre- 
sented in (10.6), we can identify the points v,’s as basic feasible solu- 
tions. However, point o fails to meet the equality constraint x, + x, + 
x, = 1, making it ineligible as a basic solution. On the other hand, 
point w is feasible but does not qualify as basic according to Defi- 
nition 10.5. Nevertheless, if we replace the equality constraint x, + 


Figure 10.11 The 
polyhedron given in 
X, +X, = 1 with the inequality constraints x,+x,+x,<1,theno Example 10.18 with 


transforms into a basic feasible solution, as shown in Figure 10.11. four corners. 
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Figure 10.12 Basic solutions and basic feasible solutions. 


Example 10.19 In Figure 10.12, the points a, b,c, d,e,f, and g all represent basic solu- 
tions since they each have two linearly independent constraints that are active. Specifically, 
points a,b,d,e, and f are considered basic feasible solutions as they fulfill all imposed 
constraints. E 


We give, without proof, the following result in this context. For a proof, see, for example, 
Bertsimas and Tsitsiklis (1997). 


Theorem 10.1 Let x* be a point in a nonempty polyhedron P. Then the following are 
equivalent: 


(a) x* is a vertex. 
(b) x* is an extreme point. 
(c) x* isa basic feasible solution. 


Definition 10.6 Two distinct basic solutions to a set of linear constraints in R” are called 
adjacent if there are n — 1 linearly independent constraints that are binding at both of them. 


As an example, in Figure 10.12, the points a and g are adjacent to the point b, and the 
points d and e are adjacent to f. 

In the subsequent development, we will see that we find an optimal corner point ofan LP 
problem by moving from one basic feasible solution to an adjacent basic feasible solution 
that improves the objective function value, and so on, repeating this step until we cannot 
go to an adjacent basic feasible solution that improves the objective function value. 

Let n and m be positive integers such that m < n. Let also b e R” and A € R™” with 
rank(A) = m (i.e., A has a full-row rank). The set P = {x € R” : Ax = b,x > 0} is called a 
polyhedron in standard form. Note that the number of equality constraints in P is m. 
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10.4.2 Finding Basic Feasible Solutions 


The question that arises now is, how to find basic solutions of polyhedra in standard 
form? The system Ax = b gives m linearly independent constraints as rank(A) = m. 
Consequently, we need n — m more binding constraints from x > 0 (this is n nonnegativity 
constraints: x, > 0,x, > 0,...,x, 2 0). Which n — m (out of those n) constraints to select 
for our purpose? We cannot choose any (n — m) x; s. Theorem 10.2 helps in this task. Before 
this theorem, we give some definitions. 

As a matter of notation, we use “;” for adjoining vectors and matrices in a column, and 
use “,” or “:” for adjoining them in a row. 

We write AasA = [a, : a, : --- : a,] where a, is the jth column of A. Since rank(A) = m, 
there exists an invertible matrix 


Ag = [a;, : ag ‘ta, JER™™, (10.7) 


2 


Let B ê {B,,B,,...,B,,}andN Ê {1,2,...,n} — B. We can permute the columns of A so that 
A = [Ag : Ay]. We can write the system Ax = b as Apx, + AyXy = b where x = (xp; Xy) 
(equivalently, x" = (x), x1). 


Definition 10.7 The m x m nonsingular matrix Ag, is called a basis matrix. The vector x, 
is called a basic solution (also called the vector of basic variables). The vector xy is called a 
nonbasic solution (also called the vector of nonbasic variables). 


We are now ready to state the following theorem which will be given without proof. For 
a proof, see, for example, Bertsimas and Tsitsiklis (1997). 


Theorem 10.2 Let b e R” and A € R”™*" have linearly independent rows. Consider the 
constraints Ax = b and x > 0. A vector x € R” is a basic solution if and only if we have 


(a) The columns of A, are linearly independent. 
(b) xy = 0. 


Since A, is nonsingular, we can solve the system of m linear equations Ax = b for xp. 
The solution is given by xy = 0 and x, = A;'b. The three-step procedure in the following 
workflow, followed by an example, will teach us how to construct such basic solutions. 


Workflow 10.4 We construct all basic solutions to a standard form polyhedron 
P= {x € R” : Ax = b,x > 0} by following three steps: 


(i) Choose m linearly independent columns aga); 4go)» --- » Agm): 
Gi) Set xy = 0. 
(iii) Calculate x, = A;"b. If xz > 0, then the vector x = (xg;Xy) is a basic feasible solution. 
Otherwise, x = (%,;X,) is a basic solution. 


It is clear that the maximum number of basic feasible solutions is ( K } Note that, gener- 


ally, not all of ( 4 ) choices of m columns may produce a basis (i.e., a nonsingular matrix Ap). 
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Hence, the number of basic solutions may be smaller than ea Note also that not all of 


these ( a bases may lead to basic feasible solutions. 
In the following example, which is due to Krishnamoorthy (2023a), we have that n = 5 
and m = 3, and that each of ( i ) = 10 choices produces a basic solution. 


Example 10.20 Consider the linear system 


X,+ X, 22, 
3x, + xX, 24, 
3x, +2x, < 10, 
xX, *,20. 


(10.8) 


The resulting polyhedron is shown in Figure 10.13. In the standard form, we have 


X,+ X% X =2, 
3x t+ Xx, =X; =4, 
3x, + 2x, +x, = 10, 


X Xz X3, Xy Xx, 20. 


Consequently, the following arrays draw the resulting polyhedron. 


1 1 -l 0 0 2 
A=|3 1 0 -1 O|, andb=| 4 
3 2 0 0 1 10 


The columns of A are 


1 1 =i 0 0 
a, =|3|, a, =]1}], a, =| 0 |, a, =|-1], anda, =]0}. 
3 2 0 0 I 


Figure 10.13 The polyhedron in Example 10.20. 
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Choosing B = {1, 2,3} (hence N = {4,5}) gives 


1 1 -1 
Ag=|3 1 0 |, and det (B) = —3 #0 (hence B is invertible). 
3 2 0 


Let xy = (x4; X5) = (0; 0). Finding Aj! and calculating x, = A;'b, we get 


Xi =2/3 
Xg=|xļ% |=| 6 
X3 10/3 


This point is a basic solution, but it is not a basic feasible solution because not all entries 
are nonnegative. This point corresponds to the vertex v, = (—2/3; 6). 
Choosing B = {2, 3,4} (hence N = {1,5}) gives 


1 —1 0 
Ag=|1 0 -1 |, and det(B) = 2 40 (hence B is invertible). 
2 0 0 


Let xy = (%4;x5) = (0; 0). Finding A;! and calculating x, = A;'b, we get 


Thus x = (0;5;3;1;0). This point is a basic feasible solution because all the entries are 
nonnegative. This point corresponds to the vertex v, = (0; 5). Table 10.2 summarizes the 
correspondences between the basic feasible solutions in the standard form polyhedron and 
the vertices visualized in Figure 10.13. a 


We want to emphasize that we can identify the basic feasible solution within the stan- 
dard form polyhedron for each corner point through a straightforward examination. In 
simpler terms, there is no need to systematically go through all possible combinations, such 
as the ( " ) choices for bases. Take, for instance, Example 10.20, where at vertex v,, the con- 
straints x, +x, > 2 and 3x, +x, > 4 are active, while 3x, + 2x, < 10 is not. Consequently, 


Table 10.2 Correspondences between the basic feasible solutions in the standard form 
polyhedron and the vertices visualized in Figure 10.13 


Vertex B det (A,) The variable x Basic feasible solution? 
vı {1,2,3} -3 (—2/3; 6; 10/3; 0; 0) x 
v, {2,3,4} 2 (0; 5; 3; 1;0) v 
v, {1,3,4} 3 (10/3; 0; 4/3; 6; 0) v 
v, {1,4,5} -1 (2; 0; 0; 2; 4) v 
v; {1,2,5} —2 (1; 1; 0; 0; 5) v 
Vs {2,3,5} 1 (0; 4; 2;0;2) v 
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in the corresponding basic feasible solutions within the standard form polyhedron, we 
have x, = x, = 0 and x, > 0. Additionally, both x, and x, are strictly positive. Therefore, 
Xp = (X,;X,;X,;) forms the corresponding basis. 

Likewise, consider vertex v, where the constraints x, +x, > 2 and 3x, +x, > 4 are not 
binding, but the constraints 3x, + 2x, < 10 and x, > 0 are active. Consequently, in the cor- 
responding basic feasible solutions within the standard form polyhedron, we find that x, 
and x, are both greater than 0, while x, equals 0. Additionally, it is worth noting that x, is 
strictly positive. Therefore, x, = (X2; X3; X4) forms the corresponding basis. 


10.4.2.1 Degeneracy 

Ata basic solution, we must have n linearly independent active constraints. However, since 
no more than n constraints can be linearly independent in an nth-dimensional space, it is 
possible for more than n active constraints to exist at a basic solution. In such cases, this 
basic solution is referred to as degenerate. 


Definition 10.8 A basic solution x € R” is called degenerate if more than n of the con- 
straints are active at x. In a nonempty polyhedron in standard form with A e€ R””, x is a 
degenerate basic solution if more than n — m of the components of x are zero. 


Example 10.21 (Example 10.20 revisited) 

Adding the constraint x, < 10/3 to System (10.8) results in three active constraints at v, (see 
Figure 10.13). Therefore, v, qualifies as a degenerate basic feasible solution. In the standard 
form, this constraint becomes x, +x, = 10/3, where x, represents the slack variable for the 
constraint x, < 10/3. In this case, with n = 6 and m = 4, and with x, = x, = x, = 0, we have 
more than n — m = 2 components of x equal to zero. a 


Degeneracy might not pose significant issues in small-scale problems, but it can introduce 
inefficiencies when dealing with large LP instances. In typical algorithms, the goal is to 
transition from one basic feasible solution to another nearby solution that either improves 
the objective function or, at the very least, maintains the current value. However, in the 
presence of degeneracy, the algorithm may cycle through several degenerate basic feasible 
solutions before finally reaching a vertex that genuinely enhances the objective function 
value. The extent of degeneracy largely hinges on how we represent the polyhedron. For 
instance, in Example 10.21, we could eliminate the constraint x, < 10/3 without altering 
the polyhedron, thereby resolving the degeneracy issue at v}. Additionally, if permissible, 
we could circumvent degeneracy by making minor adjustments to certain constraints, such 
as replacing x, < 10/3 with x, < 10/3 + 0.001. However, the feasibility of such modifica- 
tions heavily relies on the specific problem application. 


10.4.3 Pointedness 


A polyhedron is pointed if it contains no lines (a line is a straight one-dimensional figure 
formed when two points are connected with minimum distance between them, and both 
the ends extended to infinity). Figure 10.14 shows two polyhedra, one of them (namely P, ) 
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X2 X2 


xX] xX] 


P = {&xeR?: -1 <x < 1, x 2-1} P,= {eR?:-1<x,<1} 
(a) (b) 


Figure 10.14 Pointed polyhedron (a) versus nonpointed polyhedron (b). 


is pointed but the other (namely P,) is nonpointed. Note that every nonempty polyhedron 
subset of a pointed polyhedron is pointed. 

A good question to ask: Is every nonempty polyhedron pointed? We give the following 
theorem without proof. For a proof, see, for example, Bertsimas and Tsitsiklis (1997). 


Theorem 10.3 Assume that the polyhedron P = {x € R” : alx >b,i=1,...,m} is 
nonempty. Then the following are equivalent: 


(a) The polyhedron P is pointed. 
(b) The polyhedron P has at least one extreme point. 
(c) There exist n vectors out of the family a,,...,a,,, which are linearly independent. 


Note that, from Theorem 10.3, a bounded polyhedron is pointed. Similarly, the nonneg- 
ative orthant cone R? = {x € R” : x > 0} is pointed. Since any standard form polyhedron 
is a subset of the nonnegative orthant cone, it is pointed too. The following two corollaries 


are now immediate. 


Corollary 10.1 Every nonempty bounded polyhedron has at least one basic feasible 
solution. 


Corollary 10.2 Every nonempty polyhedron in standard form has at least one basic fea- 
sible solution. 


Note also that, from Theorem 10.3, every nonempty polyhedron P = {x € R” : Ax > b}, 
with A € R™®” and m < n, cannot have any basic feasible solution. 


10.4.4 Optimality 


In the above part, we have established the conditions for the existence of extreme points. 
In this part, we will see that if a nonempty polyhedron P has no corner points, then the 
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LP problem of minimizing a linear objective function over P cannot have a unique optimal 
solution. The following theorem presents the contrapositive of this statement. 


Theorem 10.4 Consider the LP problem over a polyhedron P. If P has at least one extreme 
point and there exists an optimal solution, then there exists an extreme point of P which is 
optimal. 


Proof: Let P = {x € R” : Ax > b}, and v be the optimal value of the cost c'x which we 
have assumed to be attained. Then Popt = {x € R” : Ax > b,c'x = v} contains all optimal 
solutions in P. By assumption, P,,, is a nonempty polyhedron. From Theorem 10.3, P is 
pointed. Since Ppt 


C P, Pop is pointed too. Using Theorem 10.3 again, P,,, has an extreme 
point, say x*. Since x* € Pope 


opt 
we have c'x* = v, that is, x* is optimal. T complete the 
proof, it remains to show that x* is an extreme point of P. 

Suppose, in the contrary, that x* is not an extreme point of P. Then, there exist y,z € P 
and a scalar 4 € (0,1) such that x* = Ay + (1 — A)g. Consequently, v = e'x* = Ac'y + (1 — 
A)e'z. Furthermore, since v is the optimal cost, c'y > v and e'z > v. It follows that c'y = 
c'z = v, and therefore y, z € Pont: But this contradicts the fact that x* is an extreme point 


of P,,- Thus, x* is an extreme point of P. The proof is complete. E 


A more general result than that in Theorem 10.4 is stated in the following theorem, which 
will be given without proof. For a proof, see, for example, Bertsimas and Tsitsiklis (1997). 


Theorem 10.5 Consider the LP problem over a polyhedron P. If P has at least one extreme 
point, then either the optimal cost is equal to —oo, or there exists an extreme point of P which 
is optimal. 


Theorems 10.4 and 10.5 specifically address polyhedra under the condition that they pos- 
sess at least one extreme point. But what about polyhedra that don not satisfy this condition? 
Interestingly, any LP problem, whether dealing with a polyhedron with or without extreme 
points, can be converted into an equivalent problem in standard form. This transformation 
enables us to apply Theorem 10.5, as highlighted in Corollary 10.2. This insight leads to the 
following corollary. 


Corollary 10.3 Consider the linear minimization problem over a nonempty poly- 
hedron P. Then either the optimal cost is equal to —oo, or there exists an optimal 
solution. 


Generally, Corollary 10.3 does not hold for nonlinear programming problems. For 
example, the nonlinear optimization problem 


min 1/x 
st. x>1, 


has no optimal solution, but the optimal cost is not —oo. 
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10.5 The Simplex Method 


We have introduced linear optimization problems and studied its geometry. Now, we are 
ready to introduce the simplex method. The word “simplex” is a general term of LP feasible 
region. Simplex method is used to solve LPs with any number of variables and constraints. 
The idea behind this method is to move from one basic feasible solution to an adjacent basic 
feasible solution so that the objective function value improves. 


10.5.1 Simplex Method for Maximization 


We begin by outlining the simplex method for solving LP problems with a focus on maxi- 
mization. 

The six-step procedure in Workflow 10.5, followed by Example 10.22, will teach us how 
to apply the simplex method for solving the maximization problem. First, we need the fol- 
lowing definition. 


Definition 10.9 LetA be an m x n matrix. Consider the standard form LP: 


max z=c'x 
s.t. Ax = b, (10.9) 
x>0. 
The corresponding canonical form is linear system of (m + 1) equations: 
z—c'x = 0, 
Ax = b. 


For example, the canonical form corresponding to the standard form LP: 
max z= 2x, + 3x, 
s.t. xX, + 2x, =4, 
2X, +X, +X; = 8, 
X1 X23 X3 20, 
is the system 
Z— 2x, — 3x, =0, 
x, + 2x, =4, 
2X, +X, +X; = 8. 


Workflow 10.5 (The simplex method) We solve a maximization LP problem by fol- 
lowing five steps: 


(i) Write the given LP in the standard form. 
(ii) Convert the standard form to a canonical form. 
(iii) Find a basic feasible solution for the canonical form. 
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(iv) If the current basic feasible solution is optimal, stop. If not, find which basic variable 
must become nonbasic and which nonbasic variable must become basic, and apply 
elementary row operations in order to move to an adjacent basic feasible solution with 
a higher value for the objective function. 

(v) Go to Step (iv). 


For guidance on determining which variables should change from basic to nonbasic (or 
vice versa) and on assessing the optimality, refer to Remarks 10.1-10.3 below. The examples 
in this section, except the last two, are due to Krishnamoorthy (2023b). 


Example 10.22 Use the simplex method to solve the following maximization LP. 


max z= 2x, + 3x, 


s.t. XxX, + 2x. <6, 
l 2 = (10.10) 
2x, +X, <8, 
X1» X2 >0. 
Solution 


We apply the steps in Workflow 10.5. Problem (10.10) in standard form is written as 


max Z= 2x, + 3x, 
S.t. xX, + 2x, +5, =6, 


10.11 
2x, +X, +S, =8, ( ) 
X1,X,5),8, 20. 
Problem (10.11) in the canonical form is written as 
Zz — 2X, — 3X, =0, 
X +245, =6, (10.12) 


2x, + xX +s, =8. 


The canonical variables, which correspond to the unit columns, are the variables z, s, and s,. 
Let BV denote the set of the basic variables. We select BV = {z, s1, S2}. Generally, we have 
|BV| = m+1 (m =2 in this example), and we choose z € BV always. Therefore, the BV 
contains the variable z plus m canonical variables. Let NBV denote the set of the nonbasic 
variables. Then NBV = {x,,x,}. 

Fixx, = x, = 0. System (10.12) now reads z = 0, s, = 6, and s, = 8, which are a basic fea- 
sible solution. 

Now, let us determine if the current basic feasible solution is optimal. An optimal 
solution is reached when we cannot further improve the value of z by increasing the value 
of any nonbasic variable (starting from zero). Currently, z = 2x, + 3x, = 0 as x, =x, =0 
(NBV = {x,,x,}). Increasing x, from 0 to 1 increases z from 0 to 2, while increasing x, 
from 0 to 1 increases z from 0 to 3 (we are increasing one variable at a time, while keeping 
the other nonbasic variable fixed at zero). It is more beneficial to increase x, here than 
x,. Generally, we select the nonbasic variable with the largest positive coefficient in the 
z expression to enter the basis. In the canonical form, we choose the nonbasic variable 
with the most negative coefficient in row-0 to enter the basis. The following remark 
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summarizes this discussion. We will move forward in solving Example 10.22 after this 
remark. a 


Remark 10.1 (Criterion for choosing the entering variable in maximization) 
The entering variable in a maximization LP problem is the nonbasic variable having the 
most negative coefficient in the z-row. 


Considering Remark 10.1, we designate x, as the entering variable in this step. 

With the entering variable identified, our next task is to find a new neighboring basic 
feasible solution by also determining a leaving variable. Note that we cannot increase the 
entering variable, x,, without bounds. As x, increases, s, or s, may decrease, and we must 
ensure that they remain nonnegative to maintain feasibility. 

From row 1 and row 2, with x, = 0, we get 


Row 1: 2x, +s, =6, whichimplies sı =6—-2x,>0, 
Row 2: Xx, +s, =8, which implies s,=8-—x,2>0. 


Note that s, and s, need to be nonnegative for feasibility. To keep s, > 0, we cannot increase 
x, beyond 6/2=3. To keep s, > 0, we cannot increase x, beyond 8/1=8. 

Thus, we let x, = 3 which makes s, = 0. In this step, x, is called the entering variable, and 
s; is called the leaving variable. 

The test in the following remark summarizes the above discussion. Applying this test 
guarantees that the basic solution remains feasible. We will move forward in solving 
Example 10.22 after this remark. 


Remark 10.2 (Minimum ratio test for choosing the leaving variable) For each 
constraint row that has a positive coefficient! for the entering variable, we compute the 
ratio: 


Right-hand side of the row 


Ratio = : 
Coefficient of entering variable in the row 


Among all these ratios, the nonbasic variable with smallest nonnegative ratio is the leaving 
variable. 


Note that the smallest among all the ratios computed in Remark 10.2 is the largest value 


that the entering variable can take. Going back to Example 10.22, the ratios are: 
Row 1: 2 = 3; <— The winner! 


Row 2: = =8. 


m=i NID 


Therefore, s, leaves the basis, that is, it becomes nonbasic, and the entering variable x, takes 
its place. 


1 We do not consider the row(s) with negative coefficients. 
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We use elementary row operations in order to make the entering variable basic in the row 
that the minimum ratio test meets the requirement outlined in Remark 10.2. 


Ro : Zz- 2X, — 3x =0, 

R,: X,+2xX,+ s =6, 

R, : 2x + X, + S,=8; 

ÎR, +R) > Ry: z- 3x + 5s =9, 
lp pi 1 1 £ 

zR SR: su + X% + 5S =3, 

ZR +R >R: 5x — 55+ 5 =5; 

1 7 4 ia -32 

Rot ZR, > Ro: z +35 +355 
1 A 2 1, _ 4 

R= shy Ry: X + 38 a p52 = 32 

RRi X - is +s =. 


We will complete the resolution of Example 10.22 once we address the following remark. 


Remark 10.3 (Criterion for optimality in maximization) In a maximization LP 
problem, the optimum is reached at the iteration where all the z-row coefficient of the 
non-basic variables are nonnegative. 


Note that in our example we cannot improve the value of z anymore (by making s, or s, 
basic). Hence, we have an optimal solution. The optimal solution is (x,;x,) = (10/3; 4/3) 


with the optimal 


value z = 32/3. 


10.5.2 The Full Tableau Method 


The full tableau method provides a more convenient approach for conducting the necessary 


calculations required by the simplex method. 


10.5.2.1 Simplex Tableau for Maximization 
If we have a maximization problem, the structure of the simplex tableau is as follows: 


Z x rhs 
iL cT — c] A7'A —c1A;'b 
= =] _ 
0 A;'A A;'b =X 


Here A; is defined in (10.7) and cz is the cost vector corresponding to the basic variables. We 
keep maintaining and updating the above table till we reach the optimality. Example 10.23 


resolves Example 10.22 using the simplex tableau method. 
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Example 10.23 (Example 10.22 revisited) 
Use the simplex tableau method to solve the following maximization problem. 
max z= 2x, + 3x, 
st. x, + 2x, <6, 
2X, + Xz <8, 
X15 X2 20. 
Solution 


After introducing slack variables, we obtain the standard form problem given in (10.11). 
Note that x = (0,0, 6,8) is a basic feasible solution. Hence, we have the following initial 


tableau: 
Zz xX, oS Si s, | rhs 
1 —2 =3 0 0 0 
0 1 © 1 Of 6les, 
0 2 1 0 1] 8|=s, 


Since we are maximizing the objective function, we select a nonbasic variable with the 
greatest positive reduced cost to be the one that enters the basis. Indicating the pivot element 
with a circled number, we obtain the following tableau: 


Z Xi X Si S, |rhs 
1 -V2 0 y2 0| 9 
0 V2 1 y2 of 3/=x, 
0 62) 0 -V2 1} 5/=s, 


Note that we brought x, into the basis and s, exited. We then bring x, into the basis; s, 
exits and we obtain the following tableau: 


Z Xi X% SA 53|- ThS 

1 0 4/3 1/3 |32/3|=Z 
0 0 1 2/3 -1/3| 4/3/=x, 
0 1 0 -1/3 2/3|10/3| =x, 


The reduced costs in the zeroth row of the tableau are all nonnegative, so the current 
basic feasible solution is optimal. In terms of the original variables x, and x,, this solution 
is x = (10/3; 4/3). The optimal value is z = 32/3. 

The above series of tableaux can be combined in one single table as follows: 


EROs z x, Xz Sy Sy rhs MR 
ce ah -2 -3 0 0 0 
R: 0 1 © 1 0 6 6/2 
Ri 2 1 0 1 8 8/1 
Ry+2R,>R,: 1 -1/2 0 3/2 0 9 
sR, >R,: 0 1/2 1 1/2 0 3 3/05 
SR, +R, >R,: 0 D 0 -1/2 1 5 5/15 
Ryo +3R,>R,: 1 0 0 4/3 1/3 32/3 
R,-3R,7R,: 0 0 1 2/3 -1/3 4/3 Optimal 
RRT 0 1 0 -1/3 2/3 10/3 tableau! 
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Example 10.24 Use the simplex method to solve the following LP. 
max z= 2x, -xX +X 
s.t. 3x1 +x, +x, <60, (adding sı) 
X,—-X,+2x, <10, (adding s,) 
X, +X, -X <20, (adding s3) 
X1, X2, X3 >0. 


Solution 
We have the following tableaux: 


EROs z X%, X, X, Sy S2 Ss, rhs MR 
Ret. i =l 0 0 oO 0 
R:0 3 1 1 0 0 60 60/3 
R:0 © -1 2 0 1 0 1010/1 
R:0 1 1 =1 0 0 1 2020/1 
R +2R >R :1 0 -1 3 0 2 0 20 
R -3R >R:0 0 4 -5 1 -3 0 3030/4 
RSR io a = 2 0 1 0 10 
-R +R; >R: 0 0 © -3 0 -l 1 1010/2 
Ro+ŻR >R: 1 0 0 3/2 0 3/2 1/2 25 
R -2R >R:0 0 0 1 1  -1 -2 10 Optimal 
R,+35R;>R,:0 1 0 1/2 0 1/2 1/2 15 tableau! 
IR >R: 0 0 1 -3/2 0 -1/2 1/2 5 


The optimal solution is given by (x1;x3;xX3) = (15; 5; 0), and the optimal value is z = 25. 
E 


10.5.2.2 Detecting the Existence of Alternative Optimal Solutions 

The simplex method can tell if alternative optimal solutions (i.e., infinitely many solutions) 
exist. The following remark signifies a pivotal insight into LP: A condition that opens the 
door to alternative optimal solutions that yield the same optimal value for the LP problem. 


Remark 10.4 Ifthe coefficient of a nonbasic variable in the zeroth row of the tableau is 
zero, then the LP problem has alternative optimal solutions. 


We have the following example. 


Example 10.25 Use the simplex tableau method to solve the following maximization 
problem. 
max zZz=4% +X 
s.t. 8x, +2x, <16, (adding sı) 
5x,+2x, <12, (adding s,) 
X1: X2 = 0. 
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Solution 
We have the following tableaux: 


EROs z x X Sy S rhs MR 
Rys do -4 1 0 0 o0 
R: 0 2 1 0 16 16/8=2 
R,: 0 5 2 0 1 12 12/5 = 2.4 
Ro+R >R: 1 0 0 1/2 0 8 
IR >R :0 1 1/4 1/8 0 2 This tableau 
SR, +R >R:0 0 -5/8 1 2 is optimal! 
R=>R:1 0 0 1⁄2 0 8 
R,-3R,7R: 0 1 0 1/3 -1/3 4/3 This tableau 
ÎR, >R,: 0 0 1 -5/6 4/3 8/3 is also optimal! 


In view of Remark 10.4, we have alternative optimal solutions. An optimal solution is 
given by (x,;x,) = (2; 0). Another optimal solution is given by (x,;x,) = (4/3; 8/3). The opti- 
mal value is z = 8. As an exercise for the reader, use the graphical method to reach the same 
conclusion. a 


10.5.2.3 Detecting Unboundedness 
The simplex method can be used to detect the unboundedness. The following remark tells 
us when we have an unbounded problem. 


Remark 10.5 If there is no candidate for the minimum ratio test, then the LP problem is 
unbounded. 


Example 10.26 Use the simplex tableau method to solve the following maximization 
problem. 


max Z= 2x, 

St. x,-xX, <4, (adding sı) 
—xX,+x,<1, (adding s) 
XX, 20. 


Solution 
We have the following tableaux: 
EROs z xX, X% 8S, S, rhs MR 
Rj: 1 0 -2 0 0 0 
R: 0 1 -1 1 0 4 16/8 =2 
R,: 0 -1 © 0 1 1 12/5 = 2.4 
Ro +2R >R: 1 -2 0 0 2 2 
R+R>R:0 0 0 1 1 5 TheLPis 
R >R: 0 -1 1 0 1 1 unbounded! 


10.5 The Simplex Method 


Note that there is no candidate for the minimum ratio test. In view of Remark 10.5, we 
have an unbounded LP problem. As an exercise for the reader, use the graphical method to 
reach the same conclusion. a 


10.5.2.4 Breaking Ties 
The following remark tells us how to break ties for entering or leaving variables if any. 


Remark 10.6 If there are ties for entering or leaving, we can break them arbitrarily. 


Later in this section, we will delve into more determined strategies for resolving tie- 
breakers when it comes to the selection of nonbasic variables. More specifically, we will 
refer to other remarks (Remarks 10.10 and 10.11) that provide more guidance on breaking 
ties, not only for the selection of nonbasic variables but also for deciding which variables 
should enter or leave the set of basic variables. By exploring these tie-breaking strategies, 
we aim to enhance more clarity and effectiveness of the decision-making process in LP 
problem-solving. 


Example 10.27 Use the simplex tableau method to solve the following maximization 
problem. 


max z=% +X% 
s.t. X1 +X, +x,<1, (adding sı) 
xı +2x, <1, (adding s,) 
XX2 X3 20. 
Solution 
We have the following tableaux: 


EROs Z x% X, X% S% S, rhs MR 


Rọ: 1 -1 -1 0 0 0 0 
R,:0 © 1 1 1 0 1 1 (A candidate) 


R,:0 1 2 0 1 1 1 (Another candidate!) 


0 
Ro+Rk, ~R: 1 0 0 1 1 1 Alternative 


(= oO 


R>R:0 1 #1 1 1 1 optimal 


-Rı+R >R,:0 0 -1 1 -1 1 O0 solutions! 


We have alternative optimal solutions. An optimal solution is given by (x1;xX3; X3) = 
(1; 0; 0). The optimal value is z = 1. E 


10.5.2.5 Simplex Tableau for Minimization 

Up to this point, our exploration has been centered on the simplex method as a means of 
tackling linear maximization problems. However, it is important to note that this method is 
versatile enough to be employed for solving linear minimization problems as well. When it 
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comes to linear minimization LP problems, the guidelines pertaining to entering variables 
and determining optimality are diametrically opposite to those governing maximization LP 
problems. Further elaboration on these nuances is provided in the subsequent remarks. 


Remark 10.7 (Criterion for choosing the entering variable in minimization) 
The entering variable in a minimization LP problem is the nonbasic variable having the 
most positive coefficient in the z-row. 


Remark 10.8 (Criterion for optimality in minimization) In a minimization LP 
problem, the optimum is reached at the iteration where all the z-row coefficient of the 
non-basic variables are nonpositive. 


In essence, the above remarks underscore the fundamental differences in approach 
between solving maximization and minimization LP problems using the simplex method, 
particularly when it comes to determining which variables to add to or remove from the 
basis and when to declare a solution as optimal. We have the following examples. 


Example 10.28 Use the simplex tableau method to solve the following minimization 
problem. 


min Z=-x,—-X, 


st. xX, -—X, <1, (adding s,) 
xX, + X, <2, (adding s,) 
X1» X2 >0. 
Solution 


We have the following tableaux: 


EROs Z xX, X, Sı S% rhs MR 


Roet g | 0 oO 0 
R:0 ay -1 1 0 11/l=1 
R:0 1 1 0 1 22/1=2 
R-R>R:1 0 2 -1 0 1 
R>R:0 1 -1 1 0 1 
-R +R >R:0 0 @ -1 1 11/2 
RaW >R: 1 0 0 0 -1 -2 


R,+(1/2)R, >R: 0 1 0 1/2 1/2 3/2 Optimal 
(1/2)R, >R,: 0 0 1 -1/2 1/2 1/2 tableau! 


The reduced costs in the zeroth row of the tableau are all nonpositive, so the current 
basic feasible solution is optimal. In terms of the original variables x, and x,, this solution 
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is x = (3/2; 1/2). The optimal value is z = —2. In addition, since the coefficient of the non- 
basic variable s, in the zeroth row of the last tableau is zero, we have alternative optimal 
solutions. a 


Example 10.29 Use the simplex tableau method to solve the following minimization 
problem. 


min Z=—-x,—-X, 

st.  2x,+x, <4, (adding s) 
3x, +5x, <15, (adding s,) 
X1: X3 >0. 


Solution 
We have the following tableaux: 


EROs Z xX, ä x% Sy S2 rhs MR 


Rer i | 0 0 0 
R, © 1 1 0 4 4/2=2 
R:0 3 5 0 1 15 15/3=5 
Ry-5R, >R: 1 0 1/2 -1/2 0 -2 
R > R, : 0 1/2 1/2 0 2 2/0.5=4 
—5R, +R, >R: 0 1 9 9/3.5 = 2.57 


1 
0 G) -3/2 

Ro- R >R: 1 0 0 -2/7 -1/7 -23/7 

R- R >R :0 1 © 5/7 -1/7 5/7 Optimal 


FR >R,:0 0 1 -3/7 2/7 18/7 tableau! 


The optimal solution is x = (5/7; 18/7). The optimal value is z = —23/7. a 


10.5.2.6 Problems with Nonpositive Variables and/or Free Variables 

Up to this point, we have explored the simplex method as a means to address linear opti- 
mization problems featuring variables constrained to be nonnegative. Handling scenarios 
where variables are required to be nonpositive is relatively straightforward. The approach 
involves introducing a new nonnegative variable that represents the negation of the orig- 
inal variable. In essence, if we encounter a variable, let us call it Xj, such that x; < 0, we 
substitute it with —x' while including the constraint x’ > 0. 

Similarly, when dealing with problems that include unrestricted-in-sign variables (often 
referred to as free variables), the solution approach remains uncomplicated. Here, the strat- 
egy is to introduce two fresh nonnegative variables, x; and x ’ with the constraint that their 
difference equals the original variable x,. Consequently, if we encounter an unrestricted- 
in-sign variable, denoted as x, we replace it with x, — x” and supplement the model with 
the constraints x! EA > 0. Notably, only one of x! or x” can be part of the basis in a given 
tableau, but not both. To further elucidate this, an illustrative example follows. 


329 


330 


10 Linear Programming 


Example 10.30 Use the simplex tableau method to solve the maximization LP: 


max Z=2x,+X, 

s.t. 3x +x, <6, 
xX, +x, <4, 
X >0. 


Solution 
Note that the variable x, is unrestricted-in-sign. An equivalent problem is 


max z= 2x +x, -x 

s.t. 3x, + x, — xy <6, (adding s,) 
Xx +X, — xy <4, (adding s,) 
X55, XY >0. 


We then have the following tableaux: 


EROs Zz x, x xy Si s, rhs MR 
Rj: 1-2 <1 1 0 0 0 
R,:0 © 1 =i 1 0 6 6/3=2 
R:0 1 1 -1 0 1 44/1=4 
Ro+SR,>Ry: 1 0 -1/3 1/3 2/3 0 4 
sR: > R20 1 1/3 -1/3 1/3 0 2 2/0/3)=6 
-iR +R >R :0 0 @ -2/3 -1/3 1 22/2/3)=3 
Ro+iR >R:1 0 0 G 1/22 1/2 5 
R-R>R:0 1 0 0 1/2 -1/2 1 Optimal 
ÎR >R: 0 0 1 -1 -1/2 3/2 3 tableau! 


An optimal solution is given by x, = 1 and x, = x} — x} = 3 — 0 = 3. The optimal value is 
z = 5. We point out that the columns corresponding to x, and x} are always identical but 
with opposite signs. a 


10.5.3 The Big-M Method 


Until this point, our exploration of the simplex method has centered on resolving linear 
optimization problems with inequality constraints of the form “<.” An intriguing question 
that arises is how to adapt this method to address maximization and minimization problems 
featuring inequality constraints of “>” or “=” nature. To tackle such problem types, a widely 
employed technique is known as the big-M method. Essentially, the big-M method extends 
the applicability of the simplex algorithm to encompass problems encompassing “greater- 
than” and/or “equal” constraints. 


10.5 The Simplex Method 


10.5.3.1 Problems with “Greater-than” and/or “Equal” Constraints 

In cases where we solely encounter “<” constraints, it is relatively straightforward to iden- 
tify an initial basic feasible solution, typically involving the slack variables. However, a 
critical question emerges: How can we establish an initial basic feasible solution when 
confronted with “>” and/or “=” constraints? The big-M method offers a solution to this 
conundrum by introducing artificial variables for each “>” and “=” constraints, following 


the steps in Workflow 10.6. 


Workflow 10.6 (The big-M method) We solve LP problems with “greater-than” 
and/or “equal” constraints by following six steps: 


(i) Modify constraints as needed so that all the right-hand side values are nonnegative. 
(ii) Add an artificial variable, say a,, for constraint i if it is a “>” or “=” constraint. Then 
add the nonnegativity constraint a, > 0. 
(ii) Add +Ma, to the objective function, where M is a big positive number, as follows: 
e For a maximization LP problem, add —Mq,. 
e Fora minimization LP problem, add +Ma,. 
(iv) Convert the resulting LP into the standard form by adding slack/excess variables. 
(v) Convert the LP into the canonical form and make the coefficient of a; in the zeroth 
row zero by using elementary row operations involving M. 
(vi) Operate Steps (iii)-(vi) in Workflow 10.5. 


As a direct application of Workflow 10.6, we have the following example. 


Example 10.31 Use the simplex tableau method to solve the following minimization 
problem. 


min z= 2x, + 3x, 


s.t. 2X, +X, = 4, 
X% —X >-l1, 
X1, X2 >0. 


Solution 

Our initial course of action entails the execution of the procedure outlined in Steps (i) 
through (v) as laid out in Workflow 10.6. In this sequence, our primary objective is to bring 
about a modification to the constraints of the problem. Our aim is to ensure that all the right- 
hand side values within these constraints are adjusted to be nonnegative. Consequently, 
we get 


min z= 2x, + 3x, 
s.t. 2X, +X, > 4, 
=X% +X, <1, 
X1: X3 >0. 


C 


Then, we add an artificial variable a, for constraint i if it is a “>” or “=” constraint. Then 
add a; > 0. We also add Ma, to the objective function, where M is a big positive number. 
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This yields 
min z= 2x, + 3x,+ Ma, 
st. 2x, +x, +a 24, 
-Xx +X, <1, 
X1 X2; Ay >0. 


Next, we convert the resulting LP into the standard form to get 


min z= 2x, + 3x, + Ma, 

st 24 +x, +a,-e, =4, 
—X, +X, + sS =1, 
X1, X2, S1, €13 Q1 20. 


Now, our next step involves the conversion of the LP problem into its canonical form. 
To achieve this, we employ elementary row operations to manipulate the coefficients of the 
variable a, in the zeroth row, making sure that they become zero. Following this preliminary 
step, we proceed to execute the subsequent steps, specifically Steps (iii) through (vi), as 
delineated in Workflow 10.5. 

This can be seen in the subsequent tableaux. 


EROs z x, X ey Sy a, rhs 
Rot I -2 -3 0 0 Cv) 0 Notin 
R,: 0 2 1 -1 0 1 4 canonical 
R,: 0 -1 al 0 1 O0 1 form 
Rj +MR,>R,: 1 2M-2 M-3 -M 0 0 4M In 
R, > R,: 0 © 1 -1 0 l1 4 canonical 
R>R,:0 -1 1 0 1 0 1 form 
P+ (= aOR, >R :1 0 -2 -1 0 -M+1 4 
iR 2 R,: 0 1 1/2 -1/2 0 1/2 2 Optimal 
ZR +R, >R, : 0 0 3/2 -1/2 1 1/2 3 tableau! 


The optimal solution is given by (x,;x,) = (2; 0). The optimal value is given by z = 4. m 


Example 10.32 Use the simplex tableau method to solve the following minimization 
problem. 


min Z= 2x, — 3x, 


s.t. X, + 3x, <9, 
2X, +5xX, 2-6, 
Xz 21, 
X = 0. 
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Solution 
Note that the variable x, does not have any sign restrictions, making it what is known as 
unrestricted-in-sign. We have the following tableaux: 


EROs z xi x X, S Sy å a, rhs 
R:1 -2 2 3 0 0 0 ©) 0 
R: 0 1 -1 3 1 0 0 0 9 
R,:0 -2 2 -5 0 1 0 0 6 
R: 0 0 0 1 0 0 <1 1 1 
Ry +MR,2R,:1 -2 2 M+3 0 0 -M 0 M 
R,>R,: 0 1 -i 3 1 0 0 0 9 
R,7R,:0 -2 2 -5 0 1 0 0 6 
R, > R,: 0 0 0 © 0 0 -1 1 1 
Ro- (M +3)\R >R: 1 -2 2 0 0 0 3 -M-3 -3 
R, -—3R; > R, : 0 1 -1 0 1 0 © -3 6 
R, +5R >R: 0 -2 2 0 0 1 —5 5 n 
R, >R, : 0 0 0 1 0 0 -1 1 1 
Ry-R, 2 Ry: 1 -3 3 0 -1 0 0 -M -9 
R >R :0 1/3 -1/3 0 1⁄3 0 1 -1 2 
ER +R, >R, : 0 -13 (9) 0 53 1 0 0 21 
iR +R, >R,: 0 1/3 -1/3 1 1⁄3 0 0 0 3 
Ry —9R, > Ry: 1 0 0 0 -16 -9 0 -M _ -198 O 
R +R, >R:0 0 0 0 2 3 4 -1 233P 
3R >R :0 -1 1 0 5 3 0 0 63T 
R, +R, >R,: 0 0 0 1 2 1 0 0 24M 


We contemplated an equivalent problem to establish a version of the problem with equal- 
ity constraints while ensuring that all variables involved are nonnegative. Our focus has 
shifted to the following form: 


min z= 2(x, — x) — 3x,+Ma, 


st. (x, — x1!) + 3x, +8, =9, 
/ a 

—2(x, — x1) — 5x, + S3 =6, 

x, -e +a =1, 

XX Nays 85, Cys Gy >0. 


So, the optimal solution is given by x, = x} =x] = 0 — 63 = —63, x, = 24 and e, = 23, 
hence (x1;X2) = (—63; 24). The optimal value is z = —198. E 
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10.5.3.2 Detecting Infeasibility 
The big-M method can be used to detect the infeasibility. The following remark tells us 
when we have an infeasible problem. 


Remark 10.9 If any artificial variable is basic in the optimal tableau, that is, a, > 0 for 
some i, then the LP problem is infeasible. 


As a direct application, we have the following example. 


Example 10.33 Use the simplex tableau method to solve the following minimization 


problem. 


min 
s.t. 


Solution 


Z= 3x, 

2x, +X, 26, 
3x, + 2x, =4, 
X1 X >0. 


Considering an equivalent problem with equality constraints and nonnegative variables 
only, we are interested in a problem of the form 


min 
s.t. 


z = 3x, + Ma, + Ma, 


2X, +X, +a, — & = 6, 
3X, + 2X, + a, =4, 
X1, X2, A1, Q2, &y 20. 


We then have the following tableaux: 


EROs Zz xı X% ë & a a, rhs 
Roi -3 o 0 @) ©) 0 
R,: 0 2 1-1 1 0 
R,: 0 3 2 0 0 i 
Ro +MR; +MR, > Rọ: 1 5M-3 3M -M 0 0 10M 
R>R:0 2 1-1 1 0 6 
R, +R, : 0 © 2 0 0 1 4 
Ry +(1- 2M)R, > Ry: 1 0 -M/3+2 -M 0 -5M/3+1 10M/3+40 
R -êR >R: 0 0 -1/3 -1 1 -2/3 10/3 P 
IR, >R, : 0 1 2/3 0 0 1/3 4/3 T 


The last tableau is optimal. Since a, = 10/3 > 0, the problem is infeasible. As an exercise 
for the reader, use the graphical method to reach the same conclusion. a 
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10.5.3.3 Summary of the Simplex Method Steps 
Considering all scenarios, we now summarize the above description of the simplex method. 


Workflow 10.7 (Overview of the simplex method) We solve a linear optimization 
problem by operating the following steps: 
(i) Modify constraints as needed so that all the right-hand side values are nonnegative. 
(ii) Add an artificial variable, say a;, for constraint i if it is a “>” or “=” constraint. Then 
add the nonnegativity constraint a; > 0. 

(iii) Add +Ma,; to the objective function, where M is a big positive number. 

(iv) Convert the resulting LP into the standard form by adding slack/excess variables. 

(v) Convert the LP into the canonical form and make the coefficient of a; in the zeroth 
row zero by using elementary row operations involving M. 

(vi) Find a basic feasible solution for the canonical form. 

(vii) If the current basic feasible solution is optimal, stop. If not, move to an adjacent basic 
feasible solution with a higher value for the objective function by applying elementary 
row operations and noting that: 

e Ifthe coefficient of a nonbasic variable in the zeroth row of the tableau is zero, then 
the LP has alternative optimal solutions. 
e If there is no candidate for the minimum ratio test, then the LP is unbounded. 
e If any artificial variable is basic in the optimal tableau, that is, a; > 0 for some i, 
then the LP is infeasible. 
(viii) Go to Step (vii). 


10.5.4 Anticycling 


The simplex method may encounter a phenomenon known as cycling, where it struggles 
to make progress. To address this issue and ensure that the simplex method always termi- 
nates, two anticycling rules have been developed. These rules are the lexicographic rule 
and Bland’s rule, named after Robert Bland, who discovered it in 1976. In this part, we will 
focus our discussion on Bland’s rule. Part of Exercise 10.21 targets the lexicographic rule. 
However, there are a number of good references to learn this and other anticycling rules, 
see for example (Bertsimas and Tsitsiklis, 1997, Section 3.4). 
Below we outline the pivoting rule for choosing the entering and leaving variables. 


Remark 10.10 (Pivoting rule) In the ordinary pivoting rule, we choose the entering 
variable with the most negative c, and choose the leaving variable according to the mini- 
mum ratio test. If there are ties, break them by picking the variable with the smallest index. 


Considering the above rule, we may get back to the starting tableau after some iterations 
in some LP problems, as in the following example attributed to Bertsimas and Tsitsiklis 
(1997). 


Example 10.34 Consider the following LP problem. 
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max Z= =x, — 20x, + ix; — 6X4 


s.t. ix — 8X, — X; + 9X4 <0, (adding x;) 
1 1 ‘ 
5% — 12x, — 5X, +3x, <0, (adding x) (10.13) 
X; + 6X4 <1, (adding x.) 
X1, X2, X3, X4 >0. 


If we use the simplex method to solve Problem 10.13 with the ordinary pivoting rule, we 
obtain the simplex tableau in Table 10.3. 

Note that the ending tableau is identical to the starting tableau. This means that the sim- 
plex method is cycling here! E 


Example 10.34 will be revisited in order to avoid cycling after discussing Bland’s rule. 


10.5.4.1 Bland’s Rule 
Bland’s rule (or the minimum index rule) is one of the algorithmic refinements of the sim- 
plex method to avoid cycling. 


Remark 10.11 (Bland’srule) Bland’s rule under which the simplex method for linear 
optimization terminates is as follows: 


e Choose the entering variable x, such that j is the smallest index with c; < 0. 
e Choose the leaving variable according to the minimum ratio test, and in the case of ties, 
choose the one with the smallest index. 


An illustrative example follows to elucidate Bland’s rule. 


Example 10.35 (Example 10.34 revisited) 

If we use the simplex method to solve Problem 10.13 with Bland’s rule, we obtain the 
simplex tableau in Table 10.4. 

Since we have applied Bland’s rule, the simplex method has terminated. The last tableau 
is optimal, the optimal solution is x = (1; 0; 1; 0), and the optimal value is z = 5/4. E 


The question that remains now in this context is: How to prevent cycling when we solve 
linear maximization problems? One answer stems from the following remark. 


Remark 10.12 Ifyou start with a minimization problem, say min f(x) subject to x € S, 
where f : R” > R is a function and S is a set, then an equivalent maximization prob- 
lem is max —f(x) subject to x € S. Similarly, if you start with a maximization problem, say 
max f(x) subject to x € S, then an equivalent minimization problem is min —f(x) subject 
tox eS. 
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Table 10.3 The simplex tableau of Example 10.34 


EROs z X X Xz X4 Xs Xe X rhs 

Rya. -3/4 20 -1/2 6 0 0 0 0 

R,: 0 -8 -1 9 1 0 0 0 

R: 0 1/2 -12 -1/2 3 0 1 0 0 

R: 0 0 0 1 6 0 0 1 1 

R+3R >R: 1 0 -4 -7/2 33 3 0 0 0 
4k, >R: 0 1 -32 —4 36 4 0 0 0 

-2R +R > R: 0 0 ©) 3/2 -15 -2 1 0 0 
R >R: 0 0 0 1 0 0 0 1 1 
R,+R, >R: 1 0 0 -2 18 1 1 0 0 
R,+8R, >R: 0 1 0 -84 -12 8 0 0 
IR + R, 7 0 0 1 3/8 -15/4 -1/2 1/4 0 0 
R,>R;: 0 0 0 1 0 0 0 1 1 

Ry +7R, > Ry: 1 1/4 0 0 -3 -2 3 0 0 
R >R: 0 1/8 0 1 —21/2 -3/2 1 0 0 
-ŻR, +R, > R,: 0 -3/64 1 0 1/16 -1⁄8 0 0 
-iR +R, >R: 0 -1/8 0 0 21/2 3/2 -1 1 1 
R,+16R, > R: 1 -1/2 16 0 0 -1 1 0 0 
R, +56R >R: 0 -5/2 56 1 0 © —6 0 0 
ŠR >R, : 0 -1/4 16/3 0 1 1/3 -2/3 0 0 
—56R, +R, > R: 0 5/2 —56 0 0 -2 6 1 1 
R+iR >R: 1 -7/4 44 1/2 0 0 -2 0 0 
IR >R,: 0 —5/4 28 1/2 0 1 -3 0 0 

—=R, +R, >R,: 0 1/6 -4 -1/6 1 0 @) 0 0 
R, +R, >R: 0 0 0 1 0 0 0 1 1 
R,+6R, >R: 1 -3/4 20 -1/2 6 0 0 0 0 
R,+9R, >R: 0 1/4 -8 -1 9 1 0 0 0 
3R, >R: 0 1/2 -12 -1/2 3 0 1 0 0 

R >R: 0 0 0 1 6 0 0 1 1 


w 


In essence, the conversion between minimization and maximization problems provides 
an essential equivalence within optimization. By simply negating the objective function 
and retaining the integrity of the constraints, whether starting from a minimization or 
maximization problem, an equivalent problem with the opposite optimization objective is 
established. 
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Table 10.4 The simplex tableau of Example 10.35 


EROs z xX, X Xz X4 Xs Xe X rhs 
Rọ: 1 3/4 20 -1/2 6 0 0 0 0 
R: 0 -8 -1 9 1 o 0 o 
R: 0 1/2 -12 -1/2 3 0 1 0 0 

R: 0 0 0 1 6 0 0 1 1 
R)+3R, > R: 1 0 -4 -7/2 33 3 0 0 0 
4R >R: 0 1 -32 —4 36 4 0 0 0 
-2R +R, > R,: 0 0 © 3/2 -15 -2 1 0 0 
R >R: 0 0 0 1 0 0 0 1 1 
R +R >R: 1 0 0 -2 18 1 1 0 0 
R,+8R, >R: 0 1 0 -34 -12 8 0 0 
iR >R: 0 0 1 3/8 15/4 -1/2 1/4 0 0 

R >R: 0 0 0 1 0 0 0 1 1 
R,+ ik, >R: 1 1/4 0 0 -3 -2 3 0 0 
iR >R: 0 1/8 0 1 —21/2 -3/2 1 0 0 
-ŻR, +R, > R: 0 -3/64 1 0 1/16 -1/8 0 0 
-1R +R, > R: 0 -1/8 0 0 21/2 3/2 -1 1 1 
R,+16R, > R: 1 -1/2 16 0 0 -1 1 0 0 
R,+56R, > R: 0 —5/2 56 1 0 2 —6 0 0 
ŽR >R: 0 -1/4 16/3 0 1 1/3 -2/3 0 0 
—56R, +R, > R: 0 D) -56 o o -2 6 1 1 
Ro +iR, >R: 1 0 24/5 0 0 -7/5 11/55 1⁄5 1/5 
R +R, >R: 0 0 0 1 0 0 0 1 1 
R,+4R, > R,: 0 0 —4/15 0 1 @) -11/15 1/10 1/10 
2R, > R: 0 1 -112/5 0 0 -4/5 12/5 2/5 OSS 
Ry + SR, > R: 1 0 2 0 21/2 0 3/2 5/4 5/4 
R>R: 0 0 0 1 0 0 0 1 1 
ZR >R: 0 0 -2 0 15/2 1 -1/2 3/4 3/4 

6R, +R, > R: 0 1 -24 0 6 0 2 1 1 


In consideration of Remark 10.12, to mitigate cycling in the context of maximizing c'x 
while adhering to certain constraints, we employ the simplex method and implement 
Bland’s rule to minimize —c'x under the same constraints. This approach yields an 
identical optimal solution, albeit with the optimal value of the maximization problem 
being equal to the result of the minimization problem, multiplied by —1. 
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10.5.5 Complexity 


Similar to any algorithmic method, the computational complexity of the simplex method is 
determined by the following two factors: (a) The computational complexity of each itera- 
tion. (b) The total number of iterations. 

The following theorem is known to hold (we refer to Section 3.3 in Bertsimas and Tsitsik- 
lis (1997)). It indicates that the amount of computation in each iteration of the full tableau 
method is propositional to the size of the coefficient matrix. 


Theorem 10.6 The number of arithmetic operations in each iteration of the simplex 
tableau algorithm solving Problem (10.9) is O(mn). 


Note that the estimate of the computational complexity in Theorem 10.6 refers to a single 
iteration. This complexity estimate is for both the worst-case time and the best-case time. 

In practice, the simplex method’s advantage lies in the observation that it typically con- 
verges in just O(m) iterations to discover an optimal solution. However, from a theoretical 
perspective, the method has its drawback, as this observation does not hold true for every 
LP problem. In fact, there exists a class of problems for which an exponential number of 
iterations is needed (Bertsimas and Tsitsiklis, 1997, Section 3.7). This phenomenon arises 
because the count of extreme points within the feasible set can grow exponentially with an 
increase in the number of variables and constraints. 


10.6 Duality in Linear Programming 


LP duality studies the relationships between pairs of linear programs and their solutions. 
The LP problem in the primal standard form is defined as 


min c'x 
s.t. Ax = b, (PILP) 
x > 0, 


where A € R™®”, b e R” and c € R” constitute given data, and x € R” is called the primal 
decision variable. 
The LP problem in the dual standard form is the dual of (PILP), which is defined as 


max b'y 
s.t. Aly <c, (DILP) 


where y € R” is called the dual decision variable. 


10.6.1 Lagrangian Duality and LP Duality 


Problem (DILP) can be derived from (PILP) through the usual Lagrangian approach. The 
optimization problems are classified into two classes: Constrained optimization problems 
and unconstrained optimization problems. This classification is based on whether or not 
we have constraints on the variables. The Lagrangian approach is a technique by which 
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Table 10.5 Correspondence rules 
between primal and dual linear 


programs 

Primal Minimum | Maximum Dual 
C >b >0 V 

N <b <0 A 

S =b urs R 

Vv >0 <c C 

A <0 >c N 

R urs =c S 


a constrained optimization problem becomes an unconstrained optimization problem by 
adding Lagrangian multipliers for the equality constraints. The Lagrangian function is a 
function that combines the objective function being optimized with functions penalizing 
constraint violations linearly. 

The Lagrangian function for (PILP) is defined as 


L(x, A, v) £ e'x — A! (Ax — b) — v'x. 


The vectors 4 and v are called Lagrangian multipliers. The dual of (PILP) has the objective 
function 


q(A, v) Ê inf L, 4, v) = A'b +inf (c — A'A — v)'x. 


The dual problem is obtained by maximizing q(å, v) subject to v > 0. 

Ifc — A'A -— v £0, the infimum is clearly —co. So we can exclude A for which c — AlA- 
v 4 0. When c — A'A — v = 0, the dual objective function is simply A'b. Hence, we can write 
the dual problem as follows: 


max b'i 
st. AlAtve=e, (10.14) 
v>0. 


Replacing 4 and v in (10.14) by x and z, respectively, we get (DILP). 

In the realm of linear optimization, it is essential to acknowledge that there exist 
diverse formulations beyond the standard forms (PILP) and (DILP). Linear optimization 
problems can take on various alternative forms to suit specific problem requirements and 
constraints. When dealing with LPs that adopt different formulations, Table 10.5 offers 
a valuable resource. This table provides a summary of the correspondence rules that 
establish the relationships between the primal and dual LPs. In other words, it outlines 
how the parameters and components of the primal and dual formulations are interrelated, 
facilitating the translation and understanding of LP problems in their various forms. 

In light of Table 10.5, we have the following remark. 


10.6 Duality in Linear Programming 


Remark 10.13 The following are three typical pairs of primal and dual LP problems: 


min cx max b'y 
(PILP) s.t. Ax=b, s.t. Aly <e, (DILP) 
x > 0; 
min cx max b'y 
(PILP) s.t. Ax>b; s.t. Aly=e, (DILP) 
y> 0, 
min c'x max b'y 
(PILP) s.t. Ax >b, s.t. Ay<e, (DILP) 
x > 0; y>0. 


The dual of the dual is the primal (see Proposition 10.1), so it does not matter which 
problem is called the primal. 


Example 10.36 The following is a pair of primal—dual linear programs. 


max 5x, + 4x — 3x3 min 4y, + Sy, 
s.t. X — 5x; 2 4, s.t. yı + 3y, 2 5, 
3X, +X, + 2x3 < 5, yn = 4, 
xX, = 0,X, urs, x; > 0; —Sy, + 2y, 2 -3, 
Vi < 0,y, 2 0. 


If we take the dual of the dual, we get 


max 5Z, + 42, — 323 
s.t. Zi — 57; 2 4, 
3Z, + Z + 2%, <5, 

Zı = 0,2Z, urs, Z, = 0, 


which is the primal problem. E 
The proof of the following proposition is left as an exercise for the reader. 


Proposition 10.1 The dual of the dual is the primal. 


10.6.2 The Duality Theorem 


The duality theorem is a very powerful theoretical tool that is very useful in applications 
because it leads to an interesting class of optimization algorithms. In this part, we state and 
prove the weak and strong duality theorems for the primal-dual pair (PILP) and (DILP). 
Al the results in this part are stated for the pair (PILP) and (DILP), but we indicate that all 
these results are satisfied for any primal-dual pair, including the pair (PILP) and (DILP) as 
well as the pair (PILP) and (DILP) outlined in Remark 10.13. 

Recall that an optimization problem is called feasible if it has at least one feasible point, 
and infeasible otherwise. Recall also that an optimization problem is called unbounded if it 
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is feasible and has unbounded optimal value. More specifically, a minimization (maximiza- 
tion) problem is called unbounded if it is feasible and has the optimal cost —oo (optimal 
cost +00). 

We state the weak duality property in Theorem 10.7. 


Theorem 10.7 (Weak duality in LP) Consider the primal-dual pair (PILP) and 
(DILP). Let (PILP) and (DILP) be both feasible. If x is a feasible solution to (PILP) and y is 
a feasible solution to (DILP), then b'y <c'x. 


Proof: Note that, in (DILP), the constraint A'y < c can be written as A'y + s = c with s > 
0. It follows that c'x — b'y = (Aly + s)'x — b'y = y'Ax+s'x-—y'b = y' (Ax —b)+s'x = 
x's > 0, where the last equality follows from the constraint Ax = b stated in (PILP), and the 
inequality follows because x > 0 and s > 0. The proof is complete. a 


The following corollary is now easy to obtain. 


Corollary 10.4 Consider the primal—dual pair (PILP) and (DILP). 


(a) If (PILP) is unbounded, then (DILP) is infeasible. 
(b) If (DILP) is unbounded, then (PILP) is infeasible. 


Proof: If we prove item (a), item (b) immediately follows by a symmetrical argument. 
Suppose, on the contrary, that Problem (PILP) is feasible, with the optimal cost —oo, 
and that Problem (DILP) is also feasible. Let w be the optimal cost in (DILP). By weak 
duality, we have w < —oo. That is, w < r for all r € R, which is impossible. This means 
that (DILP) cannot have a feasible solution. This proves item (a), and hence completes 
the proof. a 


In Figure 10.15, we show visually how the duality gap between the primal and dual LP 
problems turns to zero. That is, the difference c'x — b'y becomes zero when x is an optimal 
solution to (PILP) and y is an optimal solution to (DILP). This is the essence of the strong 
duality property, which is stated below in Theorem 10.8. 


Theorem 10.8 (Strong duality in LP) Consider the primal—dual pair (PILP) and 
(DILP). Assume that (PILP) and (DILP) are both feasible. If one of (PILP) or (DILP) has a 
finite optimal solution, so does the other, and their optimal values are equal. 


; Primal gap 


i : max bly min clx z : - 
' Dual feasible Primal feasible ' 


1 (DILP) turns to zero (PILP) 1 


NT OD 


Dual opt Primal opt Objective value 


Figure 10.15 The duality gap between the primal and dual LP problems. 
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Proof: Letx andy be feasible solutions to Problems (PILP) and (DILP), respectively. Starting 
from the weak duality (as presented in Theorem 10.7), we have the inequality b'y < c'x, 
which signifies that both Problems (PILP) and (DILP) are bounded. Let z and w represent 
the optimal values of (PILP) and (DILP), respectively. Using the weak duality once more, 
we have w < z. To establish that w = z, we can consider a contradiction. Suppose, to the 
contrary, that w < z. In that case, there exists no y that satisfies the inequalities Aly<e 
and b'y > z, or equivalently 


a (10.15) 


; aT i F ; . 
we can rewrite (10.15) as A y < ĉ. Using Farkas’ lemma (Version II; see Theorem 3.16), 
there exists a vector £ satisfying 


Ak = 0, e'% <0, and £ > 0. (10.16) 


Note that the vector £ can be written as £ ê (x; a) with a + 0. This rewrites (10.16) as 


aafe a E 


To prove that a # 0, suppose the contrary, that is, a = 0. Then, from (10.17), we have 
Ax = 0, c'x < 0, and x > 0. Applying Farkas’ Lemma (Version II) once again, we find that 
there is no vector y satisfying A'y < c. This implies that Problem (DILP) is infeasible, which 
is in contradiction with our initial assumption. 

It is now evident that a # 0, and further analysis shows that “Xx > 0. Additionally, Ax — 
ac = 0, which can be written as A( 1x) = c. This implies that the vector “x is feasible for 


> 0. (10.17) 


<0, and i 
a 


(PILP). However, from (10.17), we have c'x — az < 0, so c'(=x) < z. This contradicts the 
fact that z is the optimal value of (DILP). Hence, it is confirmed that w = z. The proof is 
complete. E 


The following example, which is due to Nemhauser and Wolsey (1988), is a direct appli- 
cation of Theorem 10.8. 


Example 10.37 Consider the following primal-dual pair of problems. 


min 7x, + 2x, max 4y, + 20y, — 7y3 
s.t. —x,+2x, < 4, s.t. —y, + 5y, — 2y; 2 7, 
5x; +X, < 20, 2y, +2 — 2y3 = 2, 
—2x, — 2X, < —7, Yir- Y2Y3 < 0. 
XX, < 0; 
Let x* 4 (32; £) and y* £ (2; £; 0). One can easily see that x* and y* are feasible in 


11° 11 11° 11 
the primal and dual problems, respectively. One can also easily see that b'y* = 304 and 


cx* = 30=. Based on the strong duality property (Theorem 10.8), since b'y* = e'x*, we 
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conclude that x* and y* are optimal in the primal and dual problems, respectively, and 
their optimal value is 30. a 


It is a natural question to ask: Can Problems (PILP) and (DILP) be both infeasible? The 
following example answers this question positively. 


Example 10.38 The following primal—dual pair of problems are both infeasible. 


min xX, + 2x, max 2y, + 4y, 
s.t. x+% = 2, st y +3y, = 1, 
3x, + 3x, = 4; yı + 3y, = n 


It is not hard now to establish the following corollary. 


Corollary 10.5 Consider the primal-dual pair (PILP) and (DILP). 


(a) If (PILP) is infeasible, then (DILP) is either infeasible or unbounded. 
(b) If (DILP) is infeasible, then (PILP) is either infeasible or unbounded. 


Proof: Note that the possibility that Problems (PILP) and (DILP) could be both infeasible 
has been grounded in Example 10.38. To establish item (a), it remains to demonstrate that 
if (PILP) is infeasible and (DILP) is feasible, then (DILP) must be unbounded. Assume that 
(PILP) is infeasible, and lety be a feasible solution for (DILP). Since (PILP)is infeasible, there 
exists no x satisfying Ax = b and x > 0. Applying Farkas’ Lemma (Version I, as presented 
in Theorem 3.15), we can conclude that there exists a vector ĵ that satisfies A'p > 0 and 
b'f < 0. Now, due to the feasibility of y in (DILP), we know that ATY < c. Let us define 
Ya Ê Y — ay for a > 0. We can then observe that 


Aly, =A' O - af) =Aly-aA'p <c-aA'f<e. 
This demonstrates that y, is feasible in (DILP). Furthermore, because b'y < 0, it is clear 
that b'y, tends toward infinity as a approaches infinity: 
b'y, = b'(y — a) = b'y — ab' f —— b'¥ + œ = o. 
This implies that Problem (DILP) is unbounded, successfully proving item (a). To prove 
item (b), we can apply a symmetrical argument similar to that of item (a) and utilize Farkas’ 


lemma (Version II). We leave the proof of this part as an exercise for the reader (see Exercise 
10.11). With this, we conclude the proof. E 


Corollary 10.6 is now obvious. See also Table 10.6. 


Corollary 10.6 There are only four possibilities for the primal-dual pair (PILP) and 
(DILP). Namely: 


(a) Both (PILP) and (DILP) are feasible and their optimal values are finite and equal. 
(b) (PILP) is infeasible and (DILP) is unbounded. 

(c) (PILP) is unbounded and (DILP) is infeasible. 

(d) Both (PILP) and (DILP) are infeasible. 


10.6 Duality in Linear Programming 


Table 10.6 Possibilities for the primal and the dual linear programs 


(PILP) (PILP) (PILP) 


Finite optimum Unbounded Infeasible 


(D|LP) Finite optimum v x x 
(D|LP) Unbounded x x v 
(D|LP) Infeasible x v v 


10.6.3 Complementary Slackness 


Complementary slackness refers to the idea that for an optimal solution, the product of the 
decision variable in the primal problem and the corresponding slack variable in the dual 
problem is equal to zero. We have the following definition. 


Definition 10.10 A slack variable is a variable that is added to an inequality constraint 
to transform it into an equality. Likewise, an excess (also called surplus or negative slack) 
variable is a variable that is subtracted to an inequality constraint to transform it into an 
equality. 


Consider the pair (PILP) and (DILP) outlined in Remark 10.13. 


min cx max b'y 
(PILP) st. Ax >b, st. Aly <e, (DILP) 
x > 0; y> 0. 


Let s ê c — Ay > 0 be the vector of slack variables of f (PILP), and e £ Ax — b > 0 be the 
vector of excess variables of (DILP). Then (PILP) and (DILP) are written as 


min c'x max b'y 
s.t. Ax-e=b, st. Aly+s=ce, 
x,e > 0; y.s > 0. 


The complementary slackness conditions for LP are provided in the following theorem. 


Theorem 10.9 (Complementary slackness) Consider the primal-dual pair (PILP) 
and re If x* is an eis solution to (PILP) and y* is an optimal solution to (DILP), 
then x*s* = 0 for all i, and y*e = 0 for all j, where e* ê Ax* — band s* 2c—A'y*. 


Proof: Note that 
clx® = (ATy* + s*)Tx* 
= y*"Ax* + gk ly 
= y*"(b+ ex) 4+ st'x* = bly* +y* e* 4 skye, 
Note also that the strong duality property (Theorem 10.8) implies that c'x* = b'y*. It 


follows that y aT ek +s*'x* =0. pn x*,e*,y* and s* are all nonnegative vectors, 
we have x = 0 for alli, and y*e = 0 for all j. The proof is complete. E 
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Example 10.39 (Example 10.37 revisited) 

To see how the complementary slackness conditions hold for the primal-dual pair in 
Example 10.37, note that the slack and excess variables are s* £ (0; 0; 62) and e* £ (0;0), 
respectively. In this case, x*e* = 0 for i = 1,2, and y*s* =0 forj = 1,2,3. C] 


Complementary slackness conditions are not only used to verify optimality but also serve 
as a foundation for duality theory, which is a powerful tool for solving and interpreting LP 
problems in various real-world applications. 


10.6.4 The Dual Optimal Solution via the Primal Simplex Tableau 


Recall that, from Theorem 10.8, if the primal and dual problems are both feasible and one 
of them has a finite optimal solution, so does the other, and their optimal values are equal. 
The question that emerges at this point is: How can one determine the optimal solution of 
a dual problem using the simplex tableau of the primal problem? We address this inquiry 
by presenting the following remark. 


Remark 10.14 If we are given the simplex tableau of a primal maximization problem, 
then 


Coefficient of s; in Ro, if the ith constraint is “ < ”; 
Optimal y, = 4 —(Coefficient of e; in Rọ), if the ith constraint is “ > ”; 
(Coefficient of a; in Ry) — M, ifthe ith constraint is “ = .”” 
If we are given the simplex tableau of a primal minimization problem, then 
Coefficient of s; in Ro, if the ith constraint is “ < ”; 
Optimal y; = 4 —(Coefficient of e; in Rọ), if the ith constraint is “ > ”; 
(Coefficient of a; in Rọ) +M, ifthe ith constraint is “ = .” 
In the following example from Winston (1996), Remark 10.14 is applied. 
Example 10.40 The dual problem of the maximization LP problem 
max z= 30x, + 100x, 
s.t. X, +X, <7, 
4x, + 10x, < 40, (10.18) 


10x, > 30, 
xX, = 0,x, = 0, 
is the minimization LP problem 
min w= 7y; + 40y, + 30y, 
s.t. yı + 4y, + 10y3 > 30, 


yı + 10y, > 100, 
Yı 2 0,y, 2 0,3 < 0. 


(10.19) 


2 This also holds when the ith constraint is “>.” 
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Solving Problem (10.18) by the simplex tableau method, we obtain 


EROs X, XX, Sy Sy ez a, rhs 

Roci —30 —100 0 0 0 M 0 

B20 i ia 0 0 0 7 

2a 4 10 0 1 0 0 40 

R: 0 10 00 0 -1 1 30 

Ry —MR, > Ry : 1 —30—10M —100 0 0 M 0 —30M 
R, >R,: 0 i aa 0 0 0 7 
R,>R,: 0 4 10 0 i 0 0 40 

| ce aan 10 00 0 -1 1 30 

Ro +3 +M)R >R: 1 0 —100 0 0 -3M+3 90 
R- TA >R :0 0 11 0 1/10 -1/10 4 
R,- ÉR, >R,: 0 0 10 0 1 3/5 -3/5 28 
R, >R, : 0 1 00 0 -1/10 1/10 3 

Ri 4108, >R: 1 0 00 10 1 M-11 370 
R>R:0 0 01-1/10 3/50 -3/50 1.2 

Rss R O 0 10-1/10 1/25 -1/25 2.8 
RR 0 1 00 0-1/10 1/10 3 


The last tableau is optimal. The optimal value is z = 370, the primal optimal solution is 
(x,;X,) = (3; 2.8). According to Remark 10.14, the dual optimal solution is: 


yı = Coefficient of s4 in Rp =0; 
Y, = Coefficient of s, in Ry = 10; 
y3 = —(Coefficient of e, in Rp) =-1, 


or y; = (Coefficient of a, in Ry) -M=(M-1)-M=-1. 
To check this, note that the dual optimal value is: 
w = 7y; + 40y, + 30y; = 370, 


which is exactly the primal optimal value. a 


10.7 A Homogeneous Interior-Point Method 


Interior-point methods (Nesterov and Nemirovskii, 1994) represent a class of highly 
efficient techniques designed to solve both linear and nonlinear programming problems. 
In stark contrast to the simplex method, interior-point methods excel by moving through 
the interior of the feasible region to reach an optimal solution. Multiple interior-point 
algorithms have been devised for LP problems, offering versatile tools for optimization 
(Bertsimas and Tsitsiklis, 1997, Chapter 9). 

Among the interior-point methods, homogeneous self-dual algorithms stand out as a 
valuable method for solving both linear and nonlinear programming. This section intro- 
duces a homogeneous interior-point algorithm tailored for solving (PILP) and (DILP) prob- 
lems outlined in Section 10.6. The content presented here draws from previous work found 
in Tucker (1957) and Ye et al. (1994). 


348 


10 Linear Programming 


We define the following feasibility sets for the primal—dual pair (PILP) and (DILP). 


Fpp = {x € R” : Ax =b, x > 0}, 

Fou Ê {0,s) E R” xR” : Aly+s=c, s>0}, 
Popp = (x ER” : Ax =b, x> 0}, 

Fone = {0,9 ER" XR" : A'y+s=c, s> 0}, 
Fip = Foup X For: 


We also make the following assumptions about the primal-dual pair (PILP) and (DILP). 
Assumption 10.1 The m rows of the matrix A are linearly independent. 
Assumption 10.2 The set Ff, is nonempty. 


Assumption 10.1 is introduced for the sake of convenience. On the other hand, Assumption 
10.2 imposes the requirement that both Problem (PILP) and its dual counterpart (DILP) 
must possess strictly feasible solutions. This condition serves as a guarantee, ensuring the 
existence of strong duality within the context of the LP problem. 

The following primal-dual LP model provides sufficient conditions (but not always nec- 
essary) for an optimal solution of (PILP) and (DILP). 


Ax = b, 
Aly+s=c, 
10.20 
x's = 0, ( ) 
x,s > 0. 
The homogeneous LP model for the pair (PILP) and (DILP) is as follows: 
Ax —br =0, 
—Aly-s+cr =0, 
c'x+b'y -K =0, 
x >0, (10.21) 
S >0, 
T >0, 


The first two equations in (10.21), with t = 1, represent primal and dual feasibility (with 
x,S > 0) and reversed weak duality. So they, together with the third equation after forcing 
x = 0, define primal and dual optimal solutions. Note that homogenizing < (i.e., making it 
a variable) adds the required variable dual to the third equation, introducing the artificial 
variable x achieves feasibility, and adding the third equation in (10.21) achieves self-duality. 

One can show that x's + tK = 0 (see Exercise 10.22). The next theorem relates (10.20) to 
(10.21), and it is easily proved. Here, as defined previously, R? = {x € R” : x > 0} is the 
nonnegative orthant cone. 


Theorem 10.10 The primal-dual LP model (10.20) has a solution if and only if the homo- 
geneous LP model (10.21) has a solution 


(x*,y*,s*,c%,«*) ERE x R” x RY xR, xR, 


such that c* > 0 and x* =0. 


10.7 A Homogeneous Interior-Point Method 


The main step at each iteration of the homogeneous interior-point algorithm for solving 
(PILP) and (DILP) is the computation of the search direction (Ax, Ay, As) from the Newton 
equations defined by the following system. 


A Ax —b At =nYr,, 
—A! Ay — As +e At = NF 4 
—c’ Ax+ b' Ay — AK =nr,, (10.22) 
K At+ T Ak=yy-Tk, 
SAx +X As = yul — Xs, 


where 1 is a vector of ones with an appropriate dimension, 7 and y are two parameters, 
X 4 Diag(x) € R™" is the diagonal matrix with the vector x € R” on its diagonal, same for 
S Ê Diag(s) € R”, and 


Aly +s- Te, 


l> 


A 
r, = br — Ax, YG 


A 
F, = 


li> 


(x7 
ma (x'S+ TK). 


We state the homogeneous algorithm for (PILP) and (DILP) in Algorithm 10.1. 


cx — b'y + K, H 


Algorithm 10.1: Generic homogeneous self-dual algorithm for LP 
Input: Data in Problems (PILP) and (DILP), (x, y, s, t, K) Ê (1,0, 1,1, 1) 
Output: An approximate optimal solution to Problem (PILP) 

1: while a stopping criterion is not satisfied do 

2: | choose y,y 

3: | compute the solution (Ax, Ay, As, At, Ax) of the linear system (10.22) 
4: | compute a step length 0 so that 

x+0Ax>0 

s+0As>0 

t+ 0At>0 

k+0Ak>0 

5: | set the new iterate according to 

(x, y, S, T, K) Ê (x, y, S, T, K) + O(Ax, Ay, As, At, Ax) 
6: end 


The following theorem is known to hold (see Ye et al. (1994)). It gives the computational 
complexity (worst behavior) of Algorithm 10.1 in terms of the dimension of the decision 
variable (n). 


Theorem 10.11 Let €, > 0 be the residual error at a starting point, and e > 0 be a given 
tolerance. Under Assumptions 10.2 and 10.1, if the pair (PILP) and (DILP) has a solution 
(x*, y*, s*), then Algorithm 10.1 finds an e-approximate solution (i.e., a solution with 
residual error less than or equal to e) in at most 


O (Vain g (x* +s*) (2 ))) iterations. 


349 


350 


10 Linear Programming 
Theoretically, the advantage of this interior-point method is maintaining the iteration 
complexity of O(y/n In(L)), where L is the data length of the underlying LP problem. Practi- 


cally, the disadvantage of this method is the doubled dimension of the system of equations, 
which must be solved at each iteration. 


Exercises 


10.1 Choose the correct answer for each of the following multiple-choice question- 


s/items. 
(a) All LP problems may be solved using the graphical method. 
(i) True. (ii) False. 


(b) Al-Akhawayn Inc. manufactures two varieties of paper towels, known as 
“regular” and “super-soaker.” The marketing department has established a 
requirement that the total monthly production of regular paper towels should 
not exceed twice the monthly production of super-soaker paper towels. In 
this context, let us denote by x, the quantity of regular paper towels produced 
per month and by x, the quantity of super-soaker paper towels produced per 
month. The relevant constraint(s) can be expressed as: 

(i) 2x, <x. (iii) x, < 0.5x5. (v) x, — 0.5x, > 0. 
(ii) x, < 2x. (iv) x, -x <0. 

(c) Problem A is a given formulation of an LP problem with an optimal solution. 
Problem B is a formulation obtained by multiplying the objective function of 
Problem A by a positive constant and leaving all other things unchanged. Prob- 
lems A and B will have 

(i) the same optimal solution and same objective function value. 

(ii) the same optimal solution but different objective function values. 
(iii) different optimal solutions but same objective function value. 
(iv) different optimal solutions and different objective function values. 

(d) Consider the following LP problem: 


max 12x + 10y 


s.t. 4x + 3y < 480, 
2x + 3y < 360, 
x,y 2 0. 
Which of the following points (x, y) could be a feasible corner point? 
(i) (40, 48). (iii) (180, 120). (v) None of the above. 
(ii) (120, 0). (iv) (30, 36). 


(e) Al-Akhawayn Inc. manufactures two categories of printers, which are labeled 
as “regular” and “high-speed.” The regular printers utilize 2 units of recycled 
plastic per unit produced, while the high-speed printers consume 1 unit of recy- 
cled plastic per unit of production. The company has a monthly supply of 5000 
units of recycled plastic. To produce these printers, a critical machine is essen- 
tial, with each unit of regular printers requiring 5 units of machine time and 
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each unit of high-speed printers necessitating 3 units of machine time. The 
total available machine time per month amounts to 10,000 units. In this con- 
text, let us denote the number of units of regular printers produced per month 
as x, and the number of units of high-speed printers produced per month as x,. 
The relevant constraint(s) can be expressed as: 

(i) 2x, +x, = 5000. (iii) 5x, + 3x, < 10,000. (v) (ii) and (iii). 

(ii) 2x, +x, < 5000. (iv) (i) and (iii). 

(f) Problem A is a given formulation of an LP with an optimal solution and its 
constraint 1 is < type. Problem B is a formulation obtained from Problem A by 
replacing the < constraint by an equality constraint and leaving all other things 
unchanged. Problems A and B will have 


(i) the same optimal solution and same objective function value. 
(ii) the same optimal solution but different objective function values. 
(iii) different optimal solutions but same objective function value. 
(iv) same or different solution profile depending on the role of the constraints 
in the solutions. 
(g) Consider the following LP problem: 


max 12x + 10y 
s.t. 4x+3y <480, 
2x+3y <360, 


x,y > 0. 
Which of the following points (x,y) is not in the feasible region? 
(i) (30, 60). (iii) (0, 110). (v) None of the above. 
(ii) (105, 0). (iv) (100, 10). 


(h) In any graphically solvable LP problem, if two feasible points exist, then any 
nonnegative weighted average of these points (with weights summing up to 1) 
is also feasible. 

(i) True. (ii) False. 

(i) In a two-variable graphical LP problem, if the coefficient of one of the variables 
in the objective function is changed (while the other remains fixed), then the 
slope of the objective function expression will change. 

(i) True. (ii) False. 

(j) Al-Akhawayn Inc. engages in the production of two printer variants, denoted 
as “regular” and “high-speed.” The regular printers consume 2 units of recy- 
cled plastic per unit of production, while the high-speed printers utilize 1 unit 
of recycled plastic per unit manufactured. As part of its commitment to sustain- 
ability, Al-Akhawayn ensures that a minimum of 5000 units of recycled plastic 
are used each month. The manufacturing process requires a crucial machine, 
with each unit of regular printers demanding 5 units of machine time and each 
unit of high-speed printers necessitating 3 units of machine time. The total 
available machine time per month is limited to 10,000 units. Given this context, 
we can represent the number of units of regular printers produced per month as 
x, and the number of units of high-speed printers produced per month as x,. 
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By imposing these constraints, along with the nonnegativity constraints, we 
can identify one of the feasible corner points as (assuming the first number in 
the parenthesis is x, and the second number in the parenthesis is x, ): 

(i) (0,0). (iii) None exists. (v) (2500, 0). 

(ii) (2000, 0). (iv) (0, 5000). 

(k) Ifa graphically solvable LP problem is unbounded, then it can always be con- 

verted to a regular bounded problem by removing a constraint. 

(i) True. (ii) False. 


(1) A point that satisfies all of a problem’s constraints simultaneously is a(n): 


(i) optimal solution. 

(ii) corner point. 
(iii) intersection of the profit line and a constraint. 
(iv) intersection of two or more constraints. 

(v) None of the above. 

(m) In a two-variable graphical LP problem, if the RHS of one of the constraints 
is changed (keeping all other things fixed) then the plot of the corresponding 
constraint will move in parallel to its old plot. 

(i) True. (ii) False. 

(n) Two models of a product — Regular (x) and Deluxe (y) -are produced by a com- 
pany. An LP model is used to determine the production schedule. The formu- 
lation is as follows: 


max 50x + 60y (maximum profit) 
s.t. 8x + 10y < 800 (labor hours), 
x+y < 120 (total units demanded), 
4x+ 5y < 500 (raw materials), 
x,y >20 (nonnegativity). 


The optimal solution is x = 100, y = 0. How many units of the labor hours must 
be used to produce this number of units? 
(i) 400. (iii) 500. (v) None of the above. 
(ii) 200. (iv) 5000. 
(0) LP theory states that the optimal solution to any problem will lie at: 
(i) the origin. 
(ii) a corner point of the feasible region. 
(iii) the highest point of the feasible region. 
(iv) the lowest point in the feasible region. 
(v) none of the above. 

(p) The dual ofan LP problem with maximized objective function, all < constraints 
and nonnegative variables, has minimized objective function, all > constraints 
and nonnegative decision variables. 

(i) True. (ii) False. 


10.2 


10.3 
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(q) The two objective functions (max 5x + 7y, and min —15x — 21y) will produce 
the same solution to an LP problem. 
(i) True. (ii) False. 

(r) In order for an LP problem to have a unique solution, the solution must exist 


(i) at the intersection of the nonnegativity constraints. 
(ii) at the intersection of the objective function and a constraint. 
(iii) at the intersection of two or more constraints. 
(iv) none of the above. 


(s) Ifa minimization problem has an objective function of 2x, + 5x,, which of the 
following corner points is the optimal solution? 


(i) (0, 2). (iii) (3, 3). (v) (2,0). 
(ii) (0, 3). (iv) (1,1). 
(t) Inan LP problem with a nonempty feasible region, when the objective function 
is parallel to one of the constraints, then 


(i) the solution is not optimal. 

(ii) multiple optimal solutions may exist. 
(iii) a single corner point solution exists. 
(iv) no feasible solution exists. 

(v) none of the above. 


(u) An LP problem cannot have 


(i) no optimal solutions. 

(ii) exactly two optimal solutions. 
(iii) as many optimal solutions as there are decision variables. 
(iv) an infinite number of optimal solutions. 

(v) none of the above. 


A homemaker intends to create a blend of two food types, denoted as F, and F,, 
with the objective of ensuring that the vitamin composition of the mixture contains 
a minimum of 8 units of vitamin A and 11 units of vitamin B. The cost of Food F, 
is 60 per kg, while the cost of Food F, is 80 per kg. Food F, contains 3 units per kg 
of vitamin A and 5 units per kg of vitamin B, while Food F, contains 4 units per 
kg of vitamin A and 2 units per kg of vitamin B. Formulate this problem as an LP 
problem with the objective of minimizing the cost of the mixture. 


A baker possesses 30 ounces of flour and 5 packages of yeast. For each loaf of bread, 
5 ounces of flour and 1 package of yeast are required. The baker can sell each loaf of 
bread for 30 cents. Additionally, the baker has the option to purchase extra flour at 
a rate of 4 cents per ounce or sell any remaining flour at the same price. Formulate 
this an LP problem that can be used to assist the baker in maximizing profits, which 
are calculated as revenues minus costs. 
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10.4 A farmer owns a 126-acre farm and cultivates Radish, Onion, and Potato. When 


10.5 


10.6 


10.7 


he sells his entire harvest in the market, he earns 5 per kg for Radish, 4 per kg for 
Onion, and 5 per kg for Potato. The average yield on his farm is 1500 kg of Radish 
per acre, 1800 kg of Onion per acre, and 1200 kg of Potato per acre. To grow 100 kg 
of Radish, 100 kg of Onion, and 80 kg of Potato, he needs to spend 12.5 on water. 
The labor requirement to cultivate each crop is 6 man-days per acre for Radish and 
Potato and 5 man-days per acre for Onion. He has a total of 500 man-days of labor 
available at a rate of 40 per man-day. Write an LP model that can help the farmer 
maximize his total profit. 


Use the graphical method to solve the following LP problem. 


min z= 15x, + 10x, 
s.t. 0.25x, + X% < 65, 
1.25x, +0.5x, < 90, 
x, + x% <85, 

X, X, 20. 


A company manufactures two products, denoted as X and Y, with a combined daily 
production capacity of 9 tons. Both products, X and Y, necessitate the same produc- 
tion capacity. The company has a standing contract to deliver a minimum of 2 tons 
of X and 3 tons of Y per day to another business. The production of one ton of X 
consumes 20 machine hours, while one ton of Y requires 50 machine hours. The 
maximum daily available machine hours amount to 360. The company can sell all 
its output, and it earns a profit of JD 80 per ton for X and JD 120 per ton for Y. 

(a) Formulate this as an LP problem that can be used to maximize the total profit. 
(b) Solve this optimization problem graphically. 


A small paint company produces two paint types, labeled as P, and P,, using two 
raw materials, denoted as M, and M,. The table shown below contains the essential 
data for this scenario. 

According to a market survey, the highest daily demand for product P, is limited 
to 2 tons. Additionally, the daily demand for product P, should not surpass that of 
P, by more than 1 ton. 


Tons of raw material per ton of paints produced 


P, P, Availability 
M, 6 4 24 
M. 1 2 6 


2 


Profit per ton (in $) 500 400 


Exercises 


(a) Write an LP formulation for the problem to maximize the total daily profit 
(in $). 

(b) Solve the LP model obtained in item (a) using the graphical method. 

(c) If the number of tons to be produced for P, is restricted to be integer-valued, 
the problem obtained in item (a) is called a mixed integer program. Sketch its 
feasible region and solve it graphically. 

(d) If the number of tons to be produced for P, and P, are both restricted to be 
integer-valued, the problem obtained in item (a) becomes a pure integer pro- 
gram. Sketch its feasible region and solve it graphically. 


10.8 Use the graphical method to solve the following optimization problems. 


10.9 


10.10 


(a) min 5x+ 7y (c) max X2 
s.t. x+ 3y >6, s.t. —X,+% <1, 
5x+ 2y > 10, 3X, + 2x, < 12, 
rl 2x; +3% > 12, 
x, y0. X» X% 20, 


xX, GEZ. 


(b) max 5x+ 4y 
s.t. 6x+ 4y < 24, 
x+ 2y <6, 
=x+ y $1, 
y =2, 
x, y 20. 


Consider the following LP problem. 


max Z = 5x,+ 4x, 
s.t. 3x,+ 2x, < 12, 
x+ 2x, <6, 
—X;+ xX <1, 
X, <2, 
X> X% 20. 
Sketch the feasible region and solve it graphically for each of the following cases: 
(a) The variable x, is restricted to be integer-valued; in this case the problem 
becomes a mixed integer program. 


(b) The variables x, and x, are both restricted to be integer-valued; in this case, the 
problem becomes a pure integer program. 


Transform the following LP into the standard form. 


min z= 2x, — 4x, + 5x, — 30 


s.t. 3X, +2x, — xX, 2 10, 
—2x, + 4x3 < 35, 
4x, —X < 20, 


x 26, %<8, Xx, <10. 
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10.11 


10.12 


10.13 


Prove item (b) in Corollary 10.5. 


Find the feasible solution (x,;x,) for the original LP problem in Example 10.16 
given the feasible solution een A 5X73X3) = (45031 /3;2/3) to the same problem in 
the standard form. 


Choose the correct answer for each of the following multiple-choice questions/ 
items. 
(a) A two-variable LP problem cannot be solved by the simplex method. 
(i) True. (ii) False. 
(b) If, when we are using a simplex table to solve a maximization problem, we find 
that the ratios for determining the pivot row are all negative, then we know that 
the solution is: 


(i) unbounded. (iii) degenerate. (v) none of the above. 
(ii) feasible. (iv) optimal. 
(c) In converting a greater-than-or-equal constraint for use in a simplex table, we 
must add: 


(i) an artificial variable. 
(ii) a slack variable. 
(iii) a slack and an artificial variable. 
(iv) an excess and an artificial variable. 
(v) aslack and an excess variable. 


(d) For a minimization problem using a simplex table, we know we have reached 
the optimal solution when the row Rp: 
(i) has no numbers in it. (iv) has no nonzero numbers in it. 
(ii) has no positive numbers in it. (v) none of the above. 
(iii) has no negative numbers in it. 
(e) A feasible solution requires that all artificial variables are: 
(i) greater than zero. (iv) there are no special requirements on artificial 
(ii) less than zero. variables; they may take on any value. 
(iii) equal to zero. (v) none of the above. 


(f) If the right-hand side of a constraint is changed, the feasible region will not be 
affected and will remain the same. 
(i) True. (ii) False. 
(g) With Bland’s rule, the simplex algorithm solves feasible linear minimization 
problems without cycling when: 
(i) we choose the rightmost nonbasic column with a negative cost to select 
the entering variable. 
(ii) we choose the rightmost nonbasic column with a negative cost to select 
the leaving variable. 
(iii) we choose the leftmost nonbasic column with a negative cost to select the 
entering variable. 


Exercises 


(iv) we choose the leftmost nonbasic column with a negative cost to select the 
leaving variable. 


10.14 Use the simplex tableau method to solve the following maximization problems. 


10.15 


10.16 


(a) max z= x, +1.5x, (c) max z = 2x, — xX, +X; 
st. 2x, + 4x, < 12, s.t. 3x, +X, +x; <6, 
3x, + 2x, < 10, x, +x, + 2x, <1, 

XX 2 0. x +X% — xX <2, 


X X2, X3 > 0. 


(b) max z = 3x, + 5x, + 4x, (d) max z = 60x, + 30x, + 20x, 
s.t. 2x, + 3x7 < 8, s.t. 8x, + 6x, +X, < 48, 

2x, + 5x3 < 10, 4x, + 2x, + 1.5x; < 20, 

3x, + 2x, + 4x, < 15, 2x, + 1.5x, + 0.5x3 < 8, 
X1,Xq,X3 > 0. X <5, X,X,X; 2 0. 


Consider the maximization problem presented by the following tableau. The 
parameters a and b are unknown. 


Xita Si S  S3|rhs 
0 0 17 —3+2a 0/10 
1 0 3 —1 0| 2 
O 1 4 a 0j 2 
0 0 1 b 1| 6 


For each of the following cases, explicitly discuss how many optimal solutions, if 
any, there are to the LP problem. (If the LP is unbounded state that). 


(a) a= —2 and b = 0. (b) a = 2 and b = —1. (c) a=3/2andb=1. 


Consider the following tableau of the simplex method for a maximization LP 
problem 


X, X, Xa X, Xs X,/rhs 


va 


0 0 CG Cy GZ 


0 

0 -2 a; a, -1 O 
la, 0-3 Oa, 
0 


os NO 


(a) There have to be three basic variables. Find them and give conditions on (all or 
some of) the unknowns c}, C,, C3, A4, A3, ...,@g that make these variables basic. 

(b) Give a condition on b that makes the LP feasible and conditions on c,,c,, and 
c, that make the LP optimal. 

(c) Do we have alternative optimal solutions? Justify your answer. 
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10.17 


10.18 


10.19 


10.20 


10.21 


Consider the following optimization problem: 


max Z = 5x, -X 

s.t. x — 3x, <1, 
xXx — 4x, <3, 
X, X% 20. 


Use the simplex algorithm to show that this LP is an unbounded LP problem. 


Consider the following primal-dual pair of problems. 


min 13x, + 10x, + 6x; max 8y, + 3y, 

s.t. 5X, +X% +3x, = 8, st 5y; +3y, < 13, 
3x, +X, = 3, Yı +yz < 10, 
Xj, X%, xX = 0; 3y, <S 6. 


Show that x* £ (1;0;1) and y* £ (2; 1) are optimal in the primal and dual prob- 
lems, respectively, and find the corresponding optimal values. 


In Example 10.38, we gave a pair of problems with the property that the primal 
and dual problems are both infeasible. Give an example of another pair with this 
property. 


Consider the following LP problem. 
min Z = 5x, + 3x, — 2x; 

s.t. X, +X, +X% 24, 

2X, + 3X,—-x, 2 Y, 

xX, +3% <5, 

X: X%, X% 20. 


(a) Write down the corresponding dual LP problem. 
(b) Suppose that the simplex method has been applied directly to the primal prob- 
lem, and the resulting optimal tableau is: 


ZX X, X; @ a G a, S3 rhs 
1 -2.5 0 0 0 -M _ -1.25 1.25-M -0.75 7.5 
0 -05 0 1 0 O 0.25 —0.25 0.75 1.5 
o 0.5 1 0 0 O —0.25 0.25 0.25 3.5 
0 -1 0 0 1 -1 0 0 1 1 


(i) Deduce the optimal solution to the primal problem and the optimal value. 
(ii) Deduce the optimal solution to the corresponding dual problem. 


In this exercise, you are required to implement both the revised simplex method 
and the tableau simplex method using Octave/Matlab or another programming 
tool of your preference. Subsequently, you should conduct performance compar- 
isons between your implemented algorithms and established standard optimiza- 
tion software. To evaluate your programs, you will apply them to a selection of LP 
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problems for which you must generate random data. Finally, you are expected to 


present a well-organized and structured solution for this assignment. 


(a) 


(b) 


Write an Octave function capable of solving an LP in standard form using the 
revised simplex method. This function should accept the constraint matrix 
A, the right-hand side vector b, and the cost vector c as input and provide 
as output an optimal solution vector x along with the optimal cost. In cases 
where the LP is unbounded or infeasible, the function should appropriately 
indicate this. Additionally, the number of simplex pivots or iterations employed 
should be part of the function’s output. The function should offer flexibility 
in selecting both entering and leaving variables, with the following options 
available: 

e For choosing the entering variable, the function should provide the choice 

to implement the following options. 


— Smallest value rule: After calculating all reduced costs, choose the vari- 
able with the smallest value (i.e., the most negative reduced cost) to enter 
the basis. This should be the default option. 

— Smallest index rule/Bland’s rule: Calculate the reduced costs one at a time 
and choose the variable that first gives a negative reduced cost to enter. 
In this option, you must not calculate all reduced costs. 


e For choosing the leaving variable, the function should implement the fol- 
lowing rule: 
— Smallest index rule: From among the candidates, the variable xj with the 
smallest index j leaves. This should be the default option. 


Write an Octave function capable of solving an LP in standard form using the 
tableau simplex method. This function should accept the constraint matrix 
A, the right-hand side vector b, and the cost vector c as input and provide 
as output an optimal solution vector x along with the optimal cost. In cases 
where the LP is unbounded or infeasible, the function should appropriately 
indicate this. Additionally, the number of simplex pivots or iterations employed 
should be part of the function’s output. The function should offer flexibility 
in selecting both entering and leaving variables, with the following options 
available: 


e For choosing the entering variable, the function should provide the choice 
to implement the following options: 

— Smallest value rule: After calculating all reduced costs, choose the vari- 
able with the smallest value (i.e., the most negative reduced cost) to enter 
the basis. This should be the default option. 

- Smallest index rule/Bland’s rule: After calculating all reduced costs, 
choose the variable with the smallest index with a negative reduced cost 
to enter the basis. 

e For choosing the leaving variable, the function should provide the following 
options. 
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- Smallest index rule: From among the candidates, the variable xj with the 
smallest index j leaves. This should be the default option. 

- Lexicographic rule: The leaving variable corresponds to the lexicograph- 
ically? smallest row, after scaling (see (Bertsimas and Tsitsiklis, 1997, 
Section 3.4)). 


10.22 Use (10.21) to show that x's + tK = 0. 


Notes and Sources 


The history of optimization and LP isa blend of ancient and modern influences. The origins 
of “optimization” can be traced back to ancient civilizations, where early mathematicians 
formulated and solved various optimization problems. Early references to optimization can 
be found in the works of ancient mathematicians like Euclid and Archimedes, who sought 
to maximize or minimize certain geometric quantities. The term “calculus of variations” 
was introduced in the 18th century, with pioneers like Leonhard Euler making substantial 
contributions to the field. However, the formalization of “linear programming,” a specific 
branch of optimization, emerged in the mid-20th century. George Dantzig is often credited 
with pioneering LP during World War II, when he developed the simplex method for solving 
LP problems (refer to Dantzig (2016)). His work, along with the contributions of John von 
Neumann and Leonid Kantorovich, marked a significant turning point in the history of 
optimization and LP. 

In linear optimization problems, we optimize a linear function subject to linear equality 
and inequality constraints. In this chapter, we began our study of linear optimization with 
the graphical method. We delved into the intricacies of the geometry of LP. Subsequently, 
our focus shifted to the study of the simplex method, which is the most prevalent algorithm 
for solving linear optimization problems. After that, we delved into an exploration of the 
duality in LP. As we neared the conclusion of this chapter, we addressed the LP problems 
that extended beyond the scope of the simplex method by investigating an interior-point 
method. 

As we conclude this chapter, it is worth noting that the cited references and others, such 
as Boyd et al. (2004); Chong and Zak (2013); Nocedal and Wright (2006); Panik (1996); 
Roos et al. (1998); Schrijver (1999); Mitchell et al. (2006); Ferris et al. (2007); Griva et al. 
(2008); Aggarwal (2020); Vavasis (1999); Luenberger (1973); Bazaraa et al. (2005); Solow 
(2008); Chandru and Rao (1999); Golub and Bartels (2007); Pan (2023); Taha (1971); Hillier 
and Lieberman (2001); Peressini et al. (2012), and Ackoff and Sasieni (1968), also serve as 
valuable sources of information pertaining to the subject matter covered in this chapter. 
Exercise 10.21 is due to Krishnamoorthy (2023a). 


3 The lexicographic order, also referred to as lexical order or dictionary order, extends the concept of 
alphabetical ordering found in dictionaries to sequences of arranged symbols or, in a broader context, to 
elements within a totally ordered set. 
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In Chapter 10, we have studied linear programming. In modern convex optimization, 
the class of optimization that is an immediate conic enlargement of linear programming 
is not quadratic programming, but rather the so-called second-order cone programming. 
Second-order cone programming (SOCP for short) problems, which include linear pro- 
gramming problems and quadratic programming problems as special cases, are a class 
of convex optimization problems in which the variable is not a vector whose each of its 
components is required to be nonnegative, but rather a block vector whose each of its 
subvectors is required to reside in a second-order cone (see Definition 11.1). 

Figure 11.1 shows graphical relationships among different classes of optimization prob- 
lems. So in an SOCP problem, we optimize a linear function over the intersection of an 
affine linear manifold! with the Cartesian product of second-order cones. This chapter is 
devoted to studying SOCP problems. We also refer to Alizadeh and Goldfarb (2003) for an 
excellent survey paper on this topic. 


11.1 The Second-Order Cone and Its Algebraic Structure 


This section aims to introduce algebraic tools needed to study the SOCP problems. We start 
this by introducing notations that will be used throughout this chapter. As in Chapter 10, we 


1 A manifold, in mathematics, can be viewed as a generalization and abstraction of the notion of a curved 
surface. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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Second-order cone programming 


Convex quadratic programming 


Linear programming 


Figure 11.1 A Venn diagram of different classes of optimization problems. 


“ 


use “,” for adjoining scalars, vectors and matrices in a row, and use “;” for adjoining them in 
a column. For example, avectorx E€ R” can be written asx = (X,;X,;...5X,,). For each vector 
x € R" whose first entry is indexed with 0, we write x for the subvector consisting of entries 
1 through n — 1; therefore x = (x);%) E€ R Xx R”-1, We let E” denote the nth-dimensional real 
Euclidean space R x R”~t whose elements x are indexed from 0. 


Definition 11.1 (The second-order cone) The nth-dimensional second-order cone is 
defined as 


EY = {x € E” : x > Ilxll}, 


where ||-|| denotes the Euclidean norm. The interior of this cone is the set int E} 
{x € E” : x > |||}. 


The graph below shows the third-dimensional second-order cone E}. The cone E” is 
closed, pointed (i.e., it does not contain a pair of opposite nonzero vectors) and convex with 
nonempty interior in R”. The cone E” is one of the well-known examples of of the so-called 
symmetric cones (or self-scaled cones, see Schmieta and Alizadeh (2003); Nesterov and 
Todd (1998) for definitions). 


x0 


It is known that the space E” under the bilinear map » : E” x E” ——> F” defined as 


xly 
Xf + A ? 
forms a Euclidean Jordan algebra (see Faraut and Korányi (1994); Schmieta and Alizadeh 
(2003) for definitions) equipped with the standard inner product (x, y} = x'y. 

The spectral factorization of a given element is a factorization of this element into eigen- 
vectors together with eigenvalues. 


xoy ê | 


11.1 The Second-Order Cone and Its Algebraic Structure 


Property 11.1 (Spectral decomposition in E”) Any x € E” can be expressed in 
exactly one way as a product: 


1 1 
5 1 = is = 
x= (y+ URID(S)] # [+ Go IID] æ (11.4) 
S| eT | SST 
aw WH A(x) j 
cx) c(x) 


The following definition is based on Property 11.1. 


Definition 11.2 (Eigenvalues and eigenvectors) The decomposition in (11.1) is 
called the spectral decomposition of x € E”. The values 4%) Â Xy + [||| are called the 
eigenvalues of x, and the vectors c; ,(x) 4 (1; +X/||X||) are called the eigenvectors of x. 


Having eigenvalues, we can define spectral notions such as trace and determinant. 


Definition 11.3 (Trace and determinant) The trace and determinant of x € E” are 
defined as trace(x) £ A,(x) + A,(x) = 2x and det (x) = A,(x)A,(x) = x} — [|X I’. 


Note that x € E” (x € int E?) ifand only if det (x) > 0 (det (x) > 0). Note also that x ey = 
Strace(x oy) =x'y= (x,y). 

We call e, = (1;0) € E” the identity vector of E”. The logarithmic barrier function is 
defined on int E? as x + —Indet (x). This map will be significant in our upcoming dis- 
cussions. 

One can show that the eigenvectors c; (x) and c,(x) satisfy the properties: 


cx). cŒ) = 0, 
cx). c) = c (x), 
cx). c) = €,(%), 
cix) + ce (x) =e,. 


Any continuous function f : R —— R is defined on E” as 
F(x) È fA) + fA). 
In particular, x” for n > 2, which is defined recursively as x” £ xox"! can be redefined as 
x" ê Axe, (x) + AP (x)e,(x). 


Note that x~! ex = e,. The vector x is called invertible if x! is defined, and noninvertible 
otherwise. Note also that every positive definite element is invertible and its inverse is also 
positive definite. 


The Frobenius norm of x € E” is defined as ||x||- £ 4/420) + A3(x) = 2|lxll. 


It can be shown that, for any x,y € E”, we have 


Ieoylle < ixl] yll; < Welle Ill and lx tylle = Ixl + Ile + 4y. (11.2) 


Let x € E”. The arrow-shaped matrix is defined so that xe y = Arw(x)y for every y € E”. 


365 


366 


11 Second-Order Cone Programming 


Definition 11.4 (Arrow-shaped matrix) Associated with each vector x € E”, 


the arrow-shaped matrix of x is defined as 


~T 
Arw(x) ê E ia |. 
X xI 
Note that x € E} (x € int E") if and only if Arw(x) is positive semidefinite (Arw(x) is 


positive definite). 


Definition 11.5 (Quadratic representation matrix) Associated with each vector 


x € E”, the quadratic representation matrix of x is defined as 
zT 
A 2 2 III 2X 9X 
= 2Arw’ (x) — Arw(x*) = ; 
Qx ~) w) 2x det (x)I + 2%! 


Note that Q, is also a linear operator in E”. Note also that Arw(x)e = x, Arw(x)x = x’, 
Arw(e) = Q, = I, trace(e) = 2 and det(e)= 1 (since each of the two eigenvalues of e 
equals one). 

We say that two vectors x,y € E” are simultaneously decomposed if they share a Jor- 
dan frame, that is, x = A,(x)¢, +---+A,(x)e, andy = 40) c +: -+ 4,0)c, for a Jordan 


frame {c,,...,¢,} (hence, we have c,(x) = c,(y) for each i = 1,...,7). 
We say that two vectors x and y operator commute if for all z, we have that x o (y ° Z) = 
ye (x°z). 


It is known that two elements of E” are simultaneously decomposed if and only if they 
operator commute (Schmieta and Alizadeh, 2003, Theorem 27). 

Table 11.1 summarizes the notions associated with the algebra of the second-order cone. 

All the above notions are also used in the block sense as follows: Let x £ EEEE 0 


Y Ê Wii Y2; --- ;Y,), and x; y; € E™ for i = 1,2, ... ,r. Then 
(a) EF SE" x E" x--- x E"; ©) lll = È= lx; 
(b) E2, £ EY x EP xX- x EY; (k) fœ ê fæ); Sf æ); ; fŒ). In partic- 
(c) int E}, £ int E” x int E”? ular, x7? Ea a a 
x- xint E”; 0) eê (€4,3€n, a :e,,) is the identity 


xoy Ê e1; X,Y); 
(ce) x'y Ê Xi Xi Yi 


vector of E?; 
(m) The logarithmic barrier of x € int E”, is 


r+ 


(f) det (x) = JJ; det (x;); 
(g) trace(x) = X; trace (x;); 
h) Arw(x) ê O; Arw(x;)? ; 
O Q* Gin; 


defined as x }> — In det (x); 

(n) x and y operator commute iff x; and 
y; operator commute for each i= 
E 


Note that x has 2r eigenvalues (including multiplicities) comprised of the union of the 


eigenvalues of each x; for i = 1,2, 


2 The direct sum of two square matrices A and B is the block diagonal matrix A @ B 4 | 


rene a 


AO 
B| 


11.1 The Second-Order Cone and Its Algebraic Structure 


Table 11.1 The algebraic notions and concepts associated with the 


second-order cone. 


Algebraic notions 


Space 


Second-order cone 
Dimension 
Rank 


Inner product 
Bilinear map 
Algebra 


Identity 


Reflection matrix 
Eigenvalues 


Eigenvectors 


Spectral decomposition 
Trace 


Determinant 
Square 


Inverse 


Frobenius norm 


Arrow-shaped matrix 


Quadratic representation 


Logarithmic barrier 


Gradient of log barrier 


Hessian of log barrier 


Algebra of second-order cone 


m4ady4()-yerxR™ 
x 


En 4 {x € E" | x, > lži} 


dim(E7) =n 
rank(E", o) = 2 


xey Ê xy +Ý = xy 


x oy = x'y 
XV + Yo% 


(E”, o) 


ni | o | 
0 -I 


Ay (x) Ê x + IŽIP, 40 £ x — 1%? 


1 1 
aSa] # | 043) a 
Ill Ill 


x= A,(X)c,(x) + A,(x)c,(x) 
trace(x) Ê 2x, 


det (x) Ê x? — |||? 


eX yoy. | AP 
2X 5X 


yk a 1 = lt 
cS Tw oz) + Toe uo 


Ixl; ê V2 yx ex = V2|lxll 
ape 
Arw(x) = $ 


X Xl, 


gå | Ix? ae | 


~ | 2x,% det I, + 2" x" 


— In det (x) 

x! 

Gea Bent \|x]|? —2x,x" 
xt det 2(x) 


—2x,X det (x)I,_, +244" 


| 
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We end this section by introducing some more notations that will be used throughout 


this chapter. We write x > 0 to mean that x € E} (i.e., x is a positive semidefinite vector). 
We also write x > 0 to mean thatx € int E} (i.e.,x is a positive definite vector), and write x > 
y (x > y) to mean that x — y > 0 (x — y > 0). Note that x > 0 (x > 0) if and only if A,(x) > 
0 (A,(x) > 0) fori = 1,2,..., 7. 

We use S” to denote the set of symmetric matrices of order n, and S} to denote the set 
of symmetric positive semidefinite matrices of order n. The set S? is a convex cone and 
optimization problems over this cone will be studied in Chapter 12. Throughout this and 
next chapters, we write X > 0 to mean that X € S} (i.e., x is a positive semidefinite matrix). 
We also write X > 0 to mean that X € int S} (i.e., X is a positive definite matrix), and write 
X > Y (X > Y) to mean that X -Y > 0 (X-—Y>0). 


11.2 Second-Order Cone Programming Formulation 


In this section, we introduce the SOCP problem and formulate known classes of optimiza- 
tion problems as SOCPs. 


11.2.1 Problem Formulation 


Let r > 1 be an integer. For each i=1,2,...,r, let m, n,n; be positive integers such that 
n= pam n,. Let also x, c and z be vectors in R”, y and b be vectors in R”, and A be a matrix 
in R” such that they are all conformally partitioned as 


x F (X15Xy;...5%,), 
S Ê (8)5853...558,), 
E 
A Ê (A), A), ...,A,), 


where x;, S; c; E E” and A; E R™*" for i= 1,2, ...,r. The SOCP problem and its dual in 
multiblock structures are defined as 


min elx; +--+ + clx, max b'y 
(PISOCP,) s.t. A,x, +--:+A,x,=b, (DISOCP,) s.t. A;'y+s;=¢,i=1,...,17, 
x, 20, i=1,...,75 s > 0, i=1,...,r. 


The pair (PISOCP;, DISOCP,) can be compactly rewritten as 


min c'x max bly 
(PISOCP) s.t. Ax = b, (DISOCP) s.t. Aly+s=c, 
x> 0; s>0. 


Formulating Problems as SOCPs 


In this part, we formulate three general classes of optimization problems as SOCPs. We start 
with linear optimization. 


11.2 Second-Order Cone Programming Formulation 


11.2.1.1 Linear Programming 
The linear optimization in the standard form is the problem 


min cx + +--+ +¢,X, 
s.t. XQ, +---+x,a, = b, (11.3) 
x > 0, i=1,...,r. 
Clearly, the linear optimization problem (11.3) is Problem (PISOCP,) with n, = n, =---= 


n, = 1. In other words, when the vector x € R” resides the following Cartesian product of 
second-order cones: 


El x El x- x Ejs 
Ns ei 
r—times 
we have x > 0 (as, by definition, E} 2 {teR : t20}, and hence the SOCP problem 
reduces to a linear programming problem. 


11.2.1.2 Convex Quadratic Programming 
In convex quadratic optimization problems, we minimize a strictly convex quadratic func- 
tion subject to affine constraint functions: 


min q(x) ê x'Qx + c'x 
s.t. Ax = b, (11.4) 
x>0. 


Since Problem (11.4) is strictly convex, the matrix Q must be a symmetric positive definite 

matrix (i.e., Q = Q! and Q > O). It follows that there exists another positive definite matrix 
1 

(hence non-singular), say Q?, whose square is Q. Note that 


2 
1 i eres 1 = m 1 _ 
q(x) =x'Qx+celx = | Q? (x+ 50 te) 70.2 le = |[p||? — 70.2 le, 


where p ê Q? x+ zeve). Therefore, the quadratic optimization problem (11.4) can be 
formulated as the SOCP problem 


min yo 

s.t. Ax = b, (11.5) 
«1. 1-1 à 
J-x=;0 c, 
yz=0,x20, 


where the underlying cone is the (n + 1)st-dimensional second-order cone E”+!. Note that 
Problems (11.4) and (11.5) have the same minimizer but their optimal objective values are 
equal up to the constant iT Q le. We also point out that, more generally, convex quadrat- 
ically constrained quadratic optimization problems can also be formulated as SOCP prob- 
lems (see Alizadeh and Goldfarb (2003)). 


11.2.1.3 Rotated Quadratic Cone Programming 
Let n be a positive integer and M be a non-singular matrix of order n- 2. 
The nth-dimensional rotated quadratic cone is defined as 


K” È {x = (x94; f) E R X R x R"? : 2x,x, > IIRI, xo 20%, > 0}. 
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One can see that the rotated quadratic cone is obtained by rotating the second-order cone 
through an angle of z/6 in the x)x,-plane. We call the constraint on x that satisfies the 
inequality 2xyx, > ||%||? the hyperbolic constraint. 

In rotated quadratic cone optimization problems, a linear objective function is minimized 
subject to linear constraints and hyperbolic constraints. In fact, a rotated quadratic cone 
optimization problem can be expressed as an SOCP problem because the hyperbolic con- 
straint is equivalent to a second-order cone constraint, and this can be easily deduced by 
observing that 


x=% p EK" > 2px, > IR? 
> 4x, > —4%x, + IIR? 
=> (2% +x) > ||(2% — x; 2%)||? 


= (2X) + X13 2X) — X15 2%) € EX. 


(11.6) 


Section 11.3 contains an application with hyperbolic constraints. 


11.3 Applications in Engineering and Finance 


In this section, we describe three applications of SOCP in geometry and finance. Namely 
Euclidean facility location problem (FLP), the portfolio optimization problem with loss 
risk constraints, and the optimal covering ellipsoid problem. The material of this section 
has appeared in Lobo et al. (1998) (see also Alzalg (2012)). For more applications of SOCP, 
we refer the reader to Lobo et al. (1998); Alizadeh and Goldfarb (2003); Benson and Umit 
Seglam (2013), and Alzalg and Alioui (2022). 


11.3.1 Euclidean Facility Location Problem 


FLPs involve strategically placing new facilities while minimizing their proximity to exist- 
ing ones. FLPs manifest in various real-world scenarios, encompassing the establishment of 
airports, regional campuses, the deployment of wireless communication towers, and similar 
infrastructure planning endeavors. 

FLPs can be categorized in two ways. Firstly, based on the number of new facilities added, 
we have single FLPs when only one new facility is introduced, and multiple FLPs when 
multiple new facilities are added simultaneously. Secondly, the classification can also be 
based on the distance measurement employed. When the Euclidean distance is used, they 
are referred to as Euclidean FLPs, while if the rectilinear distance is utilized, they are termed 
rectilinear FLPs. 

In single Euclidean FLP, we are given r existing facilities represented by the fixed points 
a,,a,,...,a, in R”, and we plan to place a new facility represented by x so that we min- 
imize the weighted sum of the distances between x and each of the points a,,a,,...,a,. 
To elaborate, we tackle the following problem: 


i 
min J, w; lx- ail 
i=1 
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or, alternatively, to the SOCP model: 


r 
min > W; ti 
i=1 


s.t. (t; x — a) > 0, i = 1,2,...,7, 


where w; is the weight associated with the ith existing facility and the new facility for 
E E 

Let us consider the scenario where we need to incorporate not just one, but a total of 
m new facilities, denoted as x,,X,,...,X,, E R”. In this context, we have two distinct cases, 
contingent on whether or not there exists any interaction among these new facilities within 
the underlying model. In the absence of any interaction between these new facilities, our 
primary objective revolves around the minimization of weighted sums, representing the dis- 
tances between each new facility and the existing ones. To elaborate, we tackle the following 
SOCP model: 


m r 
min È Y o ty 


j=l i=l 


s.t. (tj: xX; —a;) > 0, i= 1,2,...,r, j = 1,2,..., m, 


where w; is the weight associated with the ith existing facility and the jth new facility for 
j= 1,2,...,m andi = 1,2,...,r. 

When there is interaction among the new facilities, our objective extends beyond the 
previous requirements. In such cases, we aim to minimize the collective sum of Euclidean 
distances between every pair of new facilities. This leads to a model formulated as follows: 


m r m j-i 
min £ > Wijtij + £ Wyn tj 
jel i=1 j=l j=l 


s.t. (tj: x; — a;) > 0, i=1,2,...,7, j = 1,2,..., m, 


sx- x) = 0, j=1,2,...,m-1, s=j+1,j+2,...,m, 


where Ô, is the weight associated with the new facilities j’ and j for j’ = 1,2,...,j— 1 


andj = 1,2,...,m. 


11.3.2 Portfolio Optimization with Loss Risk Constraints 


We consider the problem of maximizing the expected return subject to loss risk constraints, 
a renowned challenge in portfolio optimization. 

Consider a portfolio problem encompassing a collection of n assets or stocks over a prede- 
fined time horizon. In this context, we introduce the variable x,, representing the quantity 
of asset i held at the commencement of, and consistently throughout, the specified time 
frame. Additionally, we utilize p; to denote the alteration in price for asset i over this dura- 
tion. Consequently, we can represent the price vector p € R” as the set of prices during 
this period. For the sake of simplicity, we assume that p follows a Gaussian distribution 
with a known mean vector p and a covariance matrix È. As a result, the return over this 
timeframe is modeled as a Gaussian random variable denoted by r = p'x, characterized by 
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its mean F = px and variance o = x! =x, where x = (X13X>3...5X,,). It is worth noting, as 
emphasized in Lobo et al. (1998), that the selection of the portfolio variable x inherently 
involves the classical Markowitz tradeoff between the mean return and the variance of this 
random return. 

The optimization variables are the portfolio vectors x € R”. For this portfolio vector, we 
take the simplest assumption x > 0(i-e., no short position Lobo et al. (1998)) and X; x; = 1 
(i.e., unit total budget Lobo et al. (1998)). 

Given an undesired return level a and a maximum acceptable probability £ over the time 
period, our objective is to ascertain the allocation of asset i (represented as x,), denoted as 
x, which maximizes the expected return over this duration. This optimization must adhere 
to the loss risk constraint, ensuring that the probability P(r < a) < p is met throughout the 
specified period. 

As noted in Lobo et al. (1998), the constraint P(r < a) < p is equivalent to the second- 
order cone constraint 

(a-F; DOE) = 0, 
provided p < 1/2 (i.e., ®-1(f) < 0), where 

öl = en 

(z) = — e dt 
2m J- 

is the cumulative normal distribution function of a zero mean unit variance Gaussian 
random variable. To prove this (see also Lobo et al. (1998)), notice that the constraint 
P(r < a) < fp can be written as 


Since (r — r)/ yo is a zero mean unit variance Gaussian random variable, the probabil- 
ity above is simply ®((a — r)/ Vo), thus the constraint P(r < a) < p can be expressed as 
D((a — F)/ Vo) < p or (Ca — F)/ 4/0) < DB), or equivalently F + ®"1(f)\/o > a. Since 


Vo = Vala = y xy ex) = 2x], 


the constraint P(r<a)< is equivalent to the second-order cone constraint 7+ 
@~!()||Z!/? x|| > a or equivalently 
a 1 
(a-7; AE) > 0. 
Our goal is to determine the amount of the asset i (which is x; over the given period), i.e. 


determine x, such that the expected return over this period is maximized. This problem can 
be cast as an SOCP as follows: 


max px 
s.t. (a _p'x; DAE?) >0, 
x =1,x>0. 
The simple problem described above has many extensions (see Lobo et al. (1998)). 


One of these extensions is imposing several loss risk constraints, that is, the constraints 
P(r < a;i) < P, i= 1,2,...,k (where £, < 1/2, for i= 1,2, ... , k), or equivalently 
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(a; —7;01(6,) (2x) ) = 0, for i=1,2,...,k, 
to be satisfied over the period. So our problem becomes the SOCP model: 


max px 
s.t. (a, —7;071(6,)(21/?x)) > 0, i= 1,2,...,k, 
1'x=1,x > 0. 


11.3.3 Optimal Covering Ellipsoid Problem 


We consider the minimum-volume covering ellipsoid problem, a renowned challenge in 
geometric optimization. 
Suppose we have k existing ellipsoids: 


E {x € R” : x Hx + 2gix+v, <0} CR",i=1,2,...,k, 


where H; € S}, g; € R”, and v; € R are given data for i = 1,2, ...,k. We need to determine 
a ball that contains all k existing ellipsoids. We adopt the same assumptions as in Alzalg 
(2012), where we assume that the cost associated with selecting a ball comprises two ele- 
ments: 


e The center cost, which is directly proportional to the Euclidean distance between the 
center and the origin; 
e The radius cost, which is directly proportional to the square of the radius. 


The objective is to minimize the overall cost by determining both the center and radius 
of the ball. In Alzalg (2012), the author provides a concrete stochastic version of this gen- 
eral application. However, it is important to note that all the data we are considering in our 
description is deterministic. In this concrete version, we assume that n = 2. Within this con- 
text, there are fixed ellipsoids that contain targets that must be destroyed. Fighter aircraft 
take off from an origin point with a preplanned circular coverage area that encompasses 
these fixed ellipsoids. The task at hand is to optimize the size and position of this coverage 
circle to minimize the total cost. 

Our goal is to determine x € R” and y € R such that the ball B defined by 


BA {xeER" : x x— 2x x+y <0}, 
contains the existing ellipsoids £, for i = 1,2, ... , k. Notice that the center of the ball B is x, 


its radius is r ê y Xx- y, and the distance from the origin to its center is Vx'x. 
We introduce the constraints d}? > x'x and d,>r= x'x — y. That is, d, is an upper 
bound on the distance between the center of the ball 6 and the origin, Vx'x, and d, is an 


upper bound on square of the radius of the ball B. 
In order to proceed, we need the following lemma which is due to Sun and Freund (2004). 


Lemma 11.1 Suppose that we are given two ellipsoids £, c R”, i = 1,2, defined by £, = 
{x € R” : x'H.x + 2g!x + v; < 0}, where H, € S}, g; € R” and v; E€ R for i = 1,2, then £, 
contains £, if and only if there exists t > 0 such that the linear matrix inequality 
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É a a =| 
gi O g Uz 


holds. 


In view of Lemma 11.1 and the requirement that the ball 4 contains the existing ellipsoids 
£; fori = 1,2, ...,k, we accordingly add the following constraints: 


I -x H. g. 
L sa| | i=1,2,...,k, 
-x y &i Ui 


or equivalently 


tH,-I tg, +x 
M,>0, Vi=1,...,k, where M; ê rae get Bi 3 
Tigi +X TV- Y 
for each i = 1, ..., k. 
Since we are looking to minimizing d,, where d, is an upper bound on the square of the 
radius of the ball 3, we can write the constraint d, > Xx- yasd, = Xx- y. So, the matrix 


M; can be then written as 


= 7H, — I Tigi +X 
i Jagt +x" qotd, -x x] 


Now, let H; ê 2,4,2] be the spectral decomposition of H, where A; Ê Diag(Aj:; ...3 Aio), 


and let u; = 2} (tig; + X). 
Then, for each i = 1,...,k, we have 


=T — 

— E 0 =E 0 A —I u, 

M; ê +| Mi lat =|" T "eiee 
0 1 0 1 u; TD; +d, -X xX 


Consequently, M, > 0 if and only if M ; = 0 for i = 1,2,...,k. Now our formulation of the 
problem in SOCP depends on the following lemma (see also Lobo et al. (1998)). 


; = 0 if and only if t;Anin (H) 2 1 and xX < 


Lemma 11.2 For each i=1,2,...,k, i 
-5 Sin) Sj = ui /CTiAy — 1) for all j such that 7,4; > 1, 


d, + Tiv; +1's,, where s; = (Sp; Sp; -- 
and Sj =0 otherwise. 


The proof of Lemma 11.2 is left as an exercise for the reader (see Exercise 11.1). 

Now, since we are minimizing d,, then for all j = 1,2, ...,n, we can relax the definitions 
of s by replacing them by ui, < 8y(t,Ay — 1),i = 1,2, ...,k. Let c > 0 denote the cost per unit 
of the Euclidean distance between the center of the ball B and the origin, and let a > 0 be 
the cost per unit of the square of the radius of B. 

We now define the decision variable x £ (d,;d,;X;y;7). Then, by introducing the unit 
cost vector € ê (€; a; 0; 0; 0), and combining all of the above, we get the model: 
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min cx 
st. u; = El (4g, +X),i=1,2,...,k, 
ui, < 8,(tAy — D), i= 1,2,...,k, j=1,2,...,0, 
=e 
xx<o, 
o < d,+7,v,—-1's,,i=1,2,...,k, 
T; > WA (Hp. i= 1,2, ...,k, 
xx < a, 


which includes only linear and hyperbolic constraints. 


11.4 Duality in Second-Order Cone Programming 


The duality in SOCP requires the application of the Karush-Kuhn-Tucker theorem which 
we state in Theorem 11.1. We first need the following definition. 
Definition 11.6 The Lagrangian function of the general minimization problem 


min f(x) 
s.t. g(x) =0,i1=1,2,...,p, (11.7) 
h(x) < 0, j =1,2,...,q, 


is defined as 
p q 
L(x, u, v) Ê f(x) + Fugo) + Yoho). 
i=l j=l 
forx € R”,u € RẸ and v € R1. 


The Karush-Kuhn-Tucker conditions? (KKT conditions for short) are also called first- 
order necessary conditions for a point to be a local minimizer. 


Theorem 11.1 (KKT Theorem) Let x be a local minimizer for Problem (11.7), then 
there exist u € R? and v € R1 such that all the following KKT conditions are satisfied. 


V,£(x, u,v) = 0, (stationary); 

u;g;i(x) = 0, Vi, (complementary slackness); 
g(x) = 0, h(x) < 0, Vi, Vj, (primal feasibility); 

u; = 0, Vi, (dual feasibility). 


The following example is attributed to Chong and Zak (2013). 


Example 11.1 Consider the following problem. 


min xX, 
St. xX, +x, 2 2, 
=x + Xz > 0. 


3 The KKT conditions were derived independently by William Karush in 1939 and by Harold Kuhn and 
Albert Tucker in 1951. 
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The Lagrangian function is 


L(x, U,V) = XX + v (2 —X, — xXx) + v — Xp) 
XiX, — (Vy — Vz)X, — (Vy + Vz)X, + 205, 


for v,,v, € R. It follows that 
V,,£(x, u,v) = k =H | 
Xı =V — U2 


The KKT conditions are 


X, — U0, +v, = 0, 

Xx —V,—v, = 0, 

vi (2 -x — X_) + v&i — X2) = 0, 
x xX, < 0, 

2—-X,—-X, < 0, 

U1, V = 0. 


Note that the points x* = (1;1) and v* = (1;0) satisfy the KKT conditions, therefore x* is 
a candidate for being a minimizer. a 


We point out that, in Example 11.1, there is no guarantee that x* is indeed a minimizer, 
because the KKT conditions are, in general, only necessary. As a central result in convex 
programming, if Problem (11.7) is a convex program, then the KKT conditions are not only 
necessary, but also sufficient for optimality. 

Within this section, we embark on an exploration of the duality theory intricately linked 
with SOCP. We lay the foundation for this expedition by introducing the concept of the com- 
plementarity condition, a pivotal facet of SOCP’s optimality criteria. The KKT conditions 
have been introduced because they are needed for the establishment of various duality- 
related findings. Much, but not all, of the duality theory for SOCP is very similar to the 
duality theory for linear programming. 

Recall that a regular cone is self-dual if it equals its dual cone (see Definition 3.22). 
Now we need the following lemma. 


Lemma 11.3 The second-order cone E} is self-dual. 


Proof: We verify that the second-order cone E? equals its dual cone, which is defined as 


(E")* A {xe E” : x'y > 0 forall y € ane 


We first show that E? C (E")*. Let x = (%);%) € E%, we need to show x € (E")*. For any 
y € Ef, we have 


XY = xo +Y > AIDI +S > [zd] +79 > 0. 


where the first inequality follows from the fact that x,y € E}, and the second inequality fol- 
lows from Cauchy-Schwartz inequality. This implies thatx € (E")* and hence E? C (E")*. 

To prove the reverse inclusion, lety € (E%)*, we need to show thaty € E}, whichis trivial 
ify = 0. If # 0, let x = (|p|; -p) € E”. Then, we have 


X'Y =X Vy + XP =YollHll —J"Y = YollPll — IPN, 
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where we used Cauchy-Schwartz inequality, when the equality is attained, to obtain the last 
equality. Since x belongs to the second-order cone E? and y belongs to its dual, it follows 
that 


T 


0 <x!y = [PIY — II). 
As y #0, this implies that y > ||p|l. That is, y € E? and hence (E%)* C E?. The proof 


is complete. E 


Now consider Problem (PISOCP). Note that the variable x is the primal variable, and 
the variables y and s are the dual variables. We call x € E” primal feasible if Ax = b and 
x > 0. Similarly, (s, y) € E” x R” is called dual-feasible if Ay +s = cand s > 0. Note that 
the matrix A is defined to map E” into R”, and its transpose A' is defined to map R” into 
E” such that x!(A'y) = (Ax)'y. 

We state and prove the following weak duality property. 


Lemma 11.4 (Weak dualityin SOCP) Ifx and (y, s) are primal and dual feasible solu- 
tions in (PISOCP) and (DISOCP), respectively, then the duality gap is 


c'x-—b'y = x's > 0. 


Proof: Consider Problems (PISOCP) and (DISOCP). Due to their constraints, we can replace 
c with Ay + s and b with Ax and get 


c'x — b'y = (Aly + s)'x — (Ax) y = (A'y)'x + s'x — (Ax) ly = x's. 


Note that x, s € E”. By the self duality of the second-order cone E}, it concludes that x's >0 
and this completes the proof. E 


The paper Nesterov and Nemirovskii (1994) shows that the strong duality property can 
fail in general conic optimization. However, despite this, a slightly weaker property can be 
always shown in conic optimization. Now, we give conditions for such a slightly weaker 
property to hold in SOCP. We say that the primal problem is strictly feasible if there exists a 
primal feasible point £ such that £ > 0. We make the following assumption for convenience. 


Assumption 11.1 The m rows of the matrix A are linearly independent. 


Using the KKT conditions (see Theorem 11.1), we state and prove the following semi- 
strong duality result. 


Lemma 11.5 (Semi-strong duality in SOCP) Consider the primal-dual pair 
(PISOCP) and (DISOCP). If the primal problem is strictly feasible and solvable, then the 
dual problem is solvable and their optimal values are equal. 


Proof: Given the lemma’s underlying assumption, which assures the strict feasibility and 
solvability of the primal problem, we proceed with considering an optimal solution denoted 
as x, where the application of the KKT conditions becomes applicable. Consequently, this 
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entails the existence of Lagrange multiplier vectors y and s, yielding a satisfying solution 
(x, y, s) that adheres to the following set of conditions: 


Ax =b, 
Aly+s=c, 
x's = 0, 
x,s>0. 

This entails that the pair (y, s) indeed qualifies as a feasible solution for the dual prob- 
lem. Now, when considering any feasible solution pair (v, z) for the dual problem, we can 
establish the inequality b'v < c'x = s'x + b'y = b'y, utilizing the weak duality to derive 
the inequality. Subsequently, by invoking the complementary slackness, we arrive at the 


ultimate equality c'x = b'y. Thus, confirming that (y, s) stands as an optimal solution for 
the dual problem, and we attain the desired equality. This concludes the proof. a 


The following strong duality result can be obtained by applying the duality relations to 
our problem formulation. 


Theorem 11.2 (Strong duality in SOCP) Consider the primal-dual pair (PISOCP) 
and (DISOCP). If both the primal and dual problems have strictly feasible solutions, then 
they both have optimal solutions x* and ( y*, s*), respectively, and 


p* £ c'x* =d* £ b'y*(i.e., x*"s* =0). 


The following lemma describes the complementarity condition as one of the optimality 
conditions of SOCP. 


Lemma 11.6 (Complementarity condition in SOCP) Ifx,s > 0, then 


x's=0 4> xos =Q. 


Proof: For anyx,s € E” having x, s > 0, we need to show that x's = Oifand onlyifxos = 0, 
or equivalently, 


(i) x's = Xoo + (Žž, Š) = 0; (ii) x5 + sč = 0. 


The direction from right to left is straightforward since the requirement is inherently 
included within the initial assumptions. To prove the other direction, assume that 
x's = 0 (which is again (i) itself), it is enough to show that (ii) is satisfied. If x, = 0 or 
S = 0, the result is clearly trivial. So, we only consider the cases when x, > 0 and sọ > 0. 
Using Cauchy-Schwartz inequality and the fact that x,s € E”, we have 


(%, 8) = X'S > —||XI[I5I] > X50, (11.8) 


Note thatx's = Oifand only if (¥,5) = —Xx)Sp, therefore the inequalities in (11.8) are satisfied 
as equalities. This holds if and only if either x = 0 or s = 0, in which case (i) and (ii) trivially 
hold, orx + Oands £ 0,X% = —aS, where a > 0, and x, = ||X|| = a@||S|| = aso, thatis, % + aš = 
X + (Xq/S))5 = 0. This implies that x, + sọ% = 0. The proof is complete. a 


11.5 A Primal-Dual Path-Following Algorithm 


As a result of Lemma 11.6, the complementarity slackness condition for the primal 
and dual SOCP problems (PISOCP) and (DISOCP) can be equivalently represented by the 
equation xs = 0. From the above results, we get the following corollary. 


Corollary 11.1 (Optimality conditions in SOCP) Consider the primal-dual pair 
(PISOCP) and (DISOCP). Assume that both the primal and dual problems are strictly 
feasible, then (x, (y,s)) € E” x R” x E” is a pair of optimal solutions to the SOCP (PISOCP) 
and (DISOCP) if and only if 


Ax = b, 
Aly+s=c, 
xos = 0, 
x,s> 0. 


(11.9) 


We have established the duality relations in SOCP. The focus in the remaining part of this 
chapter is to solve SOCP algorithmically. 


11.5 A Primal-Dual Path-Following Algorithm 


As we mentioned earlier, interior-point methods reach a best solution by traversing the 
interior of the feasible region. There are several interior-point algorithms for SOCPs; see 
for example, Lobo et al. (1998); Alizadeh and Goldfarb (2003); Alzalg (2011b, 2014a,b, 2018, 
2020); Alzalg and Pirhaji (2017a,b), and Alzalg et al. (2019), and the references contained 
therein. 

In this section, we present a primal-dual path-following algorithm for solving SOCP prob- 
lems. The material presented in this section is based on, and similar to, any primal-dual 
path-following algorithm proposed for SOCP (see for instance Alzalg (2018)). The funda- 
mental outline of the path-following algorithms designed for solving SOCP can be summa- 
rized as follows: We commence by associating the perturbed problems with second-order 
cone programming problems (PISOCP) and (DISOCP), and then drawing a path of the cen- 
ters defined by the perturbed KKT optimality conditions. Following this path, we employ 
Newton’s method to address the corresponding perturbed equations, aiming to derive a 
search direction that facilitates descent toward the optimal solution. 

Let » > 0 be a barrier parameter. The perturbed primal problem corresponding to the 
primal problem (PISOCP) is 


min f,(x) £ c'x — p Indet (x) + ruln p 
(PISOCP,) s.t. Ax=b, 
x>0, 


and the perturbed dual problem corresponding to the dual problem (DISOCP) is 


max g,(y,s) 4 b'y + u ln det (s) —rylny 
(DISOCP,) st. Aly+s=c, 
s>0. 
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Now, we define the following feasibility sets: 


Fosoce = {x € E” : Ax =b, x > 0}, 

Posoce = {(y; 8) E R” xE” : A'y+s=c, S> 0} ; 
° A ; 7 

Peisoce N {x € E” : Ax=b, Ae M 

P Disocr F {0s € R” x E” : Aly+s =c, s>0}, 

Fsoce = Foisoce X Fpisocr: 


We also make the following assumption about the primal-dual pair (PISOCP) and 
(DISOCP). 


Assumption 11.2 The set Fà 


ocp 1S nonempty. 


Assumption 11.2 requires that Problem (PISOCP „) and its dual (DISOCP,,) have strictly 
feasible solutions, which guarantees strong duality for the SOCP problem. Note that the 
feasible region for Problems (PISOCP,,) and (DISOCP,,) is described implicitly by Fsocp: 
Due to the coercivity of the function Ía on the feasible set of (PISOCP ub Problem (PISOCP PA) 
has an optimal solution. 

The following lemma proves the convergence of the optimal solution of Problem 


(PISOCP „) to the optimal solution of Problem (PISOCP) when y approaches zero. 


Lemma 11.7 Let x, be an optimal primal solution of Problem (PISOCP,,), then 
x = lim,_,o¥,, is an optimal solution of Problem (PISOCP). 


Proof: Let fœ) £ f(x, u) and f(x) = f(x, 0). Due to the coercivity of the function Ía on the 
feasible set of (PISOCP,,), Problem (PISOCP,,) has an optimal solution, say X ys such that 


Vib %,,) = Vf, u) = 0. Then, for all x € P aspnes we have that 


= — me O a — 
fœ > f&,,n) +æ- VSE, H) + O0- OET u) 
> f(x,, 4) + ulndet x, -rulny-ry 
> c'x, — wIndet x, +ryln py + plndet x, —ryuln p -ry > e'x, — rp. 


. . 2 o 
Since x was arbitrary in Pisoce? XEF 


f œ). On the other side, we have f ,,) > min xef? Tr (x). As u goes to 0, it immediately 


ETE ; = t 
this implies that min f (x) >cx,-ru2zcecx,= 


follows that fœ) = min yere f(x). Thus, x is an optimal solution of Problem (PISOCP). 


t PISOC 
The proof is complete. a 


11.5.1 Newton’s Method and Commutative Directions 


As we mentioned, the objective function of Problem (PISOCP D is strictly convex, hence the 
KKT conditions are necessary and sufficient to characterize an optimal solution to Problem 
(PISOCP,,). Consequently, the points x, and (y,,,S,,) are optimal solutions of (PISOCP,,) and 
(DISOCP,,) respectively if and only if they satisfy the perturbed nonlinear system 


Ax =b, x>0, 
Aly+s=c, s >0, (11.10) 
XoS = pe, p> 0, 


where e = (e, Epee es) is the identity vector of E?. 


> 
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We call the set of all solutions of System (11.10), denoted by (X35 Sp) with py > 0, the 
central path. We say that a point (x,y, s) is near to the central path if it belongs to the set 
N socp(#), which is defined as 

Nsocp(H) 2 {@;y; s) = F sede x Posode : dsocp®; S) < Ou, 0 E (0,1)} > 


where 


dsocp Œ, 8) Ê ||Qups — Helle. 


Now, we can apply Newton’s method to System (11.10) and obtain the following linear 
system 


AAx = 0, 
ATAy + As = 0, (11.11) 
xe AS + Axes = ope -— xos. 


where (Ax; Ay; As) € E? x R” x E? is the search direction, y = <x's is the normalized 
duality gap corresponding to (x; y; s), and ø € (0,1) is the centering parameter. 

Note that the strict second-order cone inequalities x,s > 0 imply that dsocp¥. 8) < 
|x es — yel|,, with equality holds when x and s operator commute (Schmieta and Alizadeh, 
2003, Lemma 30). In fact, it is known that many interesting properties become apparent 
for the analysis of interior-point methods when x and s operator commute. 

Denote by C(x, s) the set of all elements so that the scaled elements operator commute. 
That is, 


C(x,s) Ê {p € E; : p™ exists, andQ,x & Q, 5 operator commute} . (11.12) 


From (Schmieta and Alizadeh, 2003, Lemma 28), the equality x es = ye holds if and only 
if the equality (Qox) ° (Qp-15) = pe holds, for any non-singular vector p in E?. Therefore, for 
any given non-singular vector p € E”, System (11.10) is equivalent to the system 


Ax =b, x>0O, 
Aly+s=c, s >0, (11.13) 
(Qx) e (QS) = we, p >o. 


Let v € E?. With respect to a non-singular vector p € E”, we define the scaling vectors v 
and v and the scaling matrix A as 


VQ, v=Q,.v, and A+Q,A. (11.14) 


The definitions in (11.14) are valid for both the single and multiple block cases. 
Using this change of variables and the fact that Q, ( E”) = E”, we conclude that System 
(11.11) is equivalent to the following Newton system 


AAx = b- Ax, 
A'Ay + AS = c- s —A'y, (11.15) 
Xo AS + Axos = ope — Xos. 


Here, the normalized duality gap is 4 = x's = “x's, In fact, 


x's = (Q,x)'Q,18=x'Q,Q, 18 = x's. (11.16) 
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Solving the scaled Newton system (11.15) yields the search direction (Ax; Ay; As). Then, 
we apply the inverse scaling to (Ax; As) to obtain the Newton direction (Ax; As). Note that 
the search direction (Ax; Ay; As) belongs to the so-called the MZ family of directions (due 
Monteiro (1997); Zhang (1998)). 

Clearly, the set C(x, s) defined in (11.12) is a subclass of the MZ family of search direc- 
tions. Our focus is on vectors p € C(x,s). We discuss the following three choices of p (see 
(Schmieta and Alizadeh, 2003, Section 3)): 


e The first one is to choose p = x~/?, which gives x = e. 
e The second one is to choose p = s!/?, which gives s = e. 
e The third choice of p is given by p = Qyir(Qus)/?)'/?, which yields Q5x = s, and 


therefore s = Q,.18 = Q,x =X. 


The first two directional choices are respectively referred to as the HRVW/KSH/M direc- 
tion (attributed to Helmberg et al. (1996); Monteiro (1997); Kojima et al. (1997)), and the 
dual HRVW/KSH/M direction. The third direction option is known as the NT direction 
(credited to Nesterov and Todd (1998)). 


11.5.2 Path-Following Algorithm 


The path-following algorithm for solving SOCP problem is formally stated in 
Algorithm 11.1. 


Algorithm 11.1: Path-following algorithm for SOCP 
Input: Data in Problems (PISOCP) and (DISOCP), k = 0, 
(x;y; s9) © Neocp (x), € > 0, 6,0 € (0,1) 
Output: An c-optimal solution to Problem (PISOCP) 
; T 
1: while x s > edo 
2: | choose p” € C (x, s®) 
3. | compute (x yO; s ) by applying scaling to (x; y®; s®) 
i —— => 

4 | set pw ê =x) s, h® £ ye — xcs and pO £ vA xk AT 
s: | compute ( Ax; Ay; As% ) by solving the scaled system (11.15) to get 
(ax®; Ay; As) a ((n® - x®oAs\ ) os; -PO A (soh) ;-ATay) 
6: | compute (Ax; Ay; As®) by applying inverse scaling to (1x0; Ay; As ) 
7: | set the new iterate according to 
(xD, yD, gD) A (xO 4 QAO; yO 4 aO AYO; s 4 OAW) 
s: | setk=k+1 
9: end 


Algorithm 11.1 selects a sequence of displacement steps {a} and centrality parame- 
ters {o} according to the following rule: For all k > 0, we take o = 1 — 6/1/r, where 
ô € [0, Vn). Alzalg (2018) discusses in Section 4 various selections for calculating the dis- 
placement step a, 


11.5 A Primal-Dual Path-Following Algorithm 


In the rest of this section, we prove that the complementary gap and the function f, 
decrease for a given displacement step. The proof of this result depends essentially on the 
following lemma. 


Lemma 11.8 Let (x;y; s) € int R? x R” x int E”, , (x;y; s) be obtained by applying scaling 
to (x; y; s), and (Ax; Ay; As) be a solution of System (11.15). Then we have 
—T 
(a) Ax As=0. 
2 —T = = 
(b) x' As + Ax s = trace(h), where h ê ope -x es such that o € (0,1) and u = x's. 


(c) x*'st = (1 — a(l — o /2)) x's, Va € R, where x’ £ ¥ + aAx and st Ê s + gAs. 
(d) xt's+ = (1 — a(1 — co /2)) x's, Va € R, where xt ê x + «Ax and st Ê s + gAs. 


Proof: By the first two equations of System (11.15), we get 
—T —T 7 —;\T 
Ax As = -Ax A'Ay = -( 45x) Ay =0. 

This proves item (a). We prove item (b) by noting that 


trace(h) = trace (ope — xos) 


trace (X° As + Axes) 


7 — = —t 
trace (xe As) + trace (Axes) = x As + Ax s, 


where we used the last equation of System (11.15) to obtain the first equality. 

Item (c) is left as an exercise for the reader (see Exercise 11.4). Item (d) follows from item 
(c) and the fact that x's = x!s (see (11.16)), and similarly that xt st = xt's*. The proof is 
complete. a 


The result of the following theorem is a special case of that in (Alzalg, 2018, Lemma 5.2). 
Theorem 11.3 Let (x; y; s) and (xt; y*; st) be strictly feasible solutions of the pair of prob- 
lems (PISOCP,,) and (DISOCP,,) with 

(xt:y*;st) = (Œ + Ax; y+aAy; s+ ads), 


where a is a displacement step and (Ax; Ay; As) is the Newton direction. Then 


(a) xt'st < x's. (b) Fa) < f,@). 


Proof: Note that 
xt! s+ = (1 -—a (1 pe = )) x's < x's, 


where the equality follows from item (d) of Lemma 11.8 and the strict inequality follows 
from (1 — a(1 — c /2)) < 1 (asa > 0 and o € (0,1)). This proves item (a). 

To prove item (b), note that fe") ~ fi) + Vif) (x* — x), and therefore fe) — 
fœ) ~ aV f(x) Ax. Since V F(X) = — Vid ,(X)AX, we have that 


F+) —f,(0) ~ —a Ax' VF (x)Ax < 0, 
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where the strict inequality follows from the positive definiteness of the Hessian matrix 
VoL, (x) (as f, is strictly convex). Thus, f(x") < f(x). The proof is complete. E 


11.5.3 Complexity Estimates 


In this part, we analyze the complexity of the proposed path-following algorithm for SOCP. 
More specifically, we prove that the iteration-complexity of Algorithm 11.1 is bounded by 


(0) (vrin e Tg) ) i 


Our proof depends essentially on the following two lemmas. 


Lemma 11.9 Let (x;y; s) € Foso x Face 
scaling to (x;y;s) with h = ove — x° s, and (Ax; Ay; As) be a solution of System (11.15). 


For any a € R, we set 


Let also (x;y; s) be obtained by applying 


(x(a); (a); s(a)) Ê (sy; S) + a(Ax; Ay; As), 
u(a) Ê Zx(a)'s(a), 
v(a) Ê x(a) o s(a) — u(a)e. 

Then 


v(a) = (1 — a)(¥es — pe) + a? Axe AS. (11.17) 


Proof: See Exercise 11.5. E 
Lemma 11.10 Let (x;y;s) € F ake x F isocr’ and (x;y;s) be obtained by applying 
scaling to (x;y;s) such that ||xes— el] < Ou, for some @ € [0,1) and y > 0. Let also 
(Ax; Ay; As) be a solution of System (11.15), h=opwe—Xes, 6,4 p||Ax eX ||p, = 
|x ° As||p. Then, we have 


6,6 eh (524.52) < Wl (11.18) 
xs > 9 Vx sS ~ 21- 0) . 
Proof: See Exercise 11.6. a 


The following theorem analyzes the behavior of one iteration of Algorithm 11.1. 
This theorem is a special case of (Alzalg, 2018, Theorem 6.1). 


Theorem 11.4 Let 0 € (0,1) and 6 € [0, Vr) be given such that 
0? + 8? 


21-0} (1- =) 


Suppose that (x; y; s) E Nocp(H) and let (Ax; Ay; As) denote the solution of System (11.15) 
with h = ove—Xesando = 1 — 6/y/r. Then, we have 


<0<-. (11.19) 


Nile 
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(a) xt st =(1- 5/\/r)x's. 
(b) Œ"; yt; s+) = Gy; 8) + (Ax; Ay; As) E€ Ngocp(H): 
(c) (xt; yt; st) = (x;y; s) + (Ax; Ay; As) E Neocp()- 


Proof: Item (a) follows directly from item (c) of Lemma 11.8 with a = 1ando = 1 — ô/ yr. 
We now prove item (b). Define 


i 
paly st=(1-2)y (11.20) 
re r 
and let (x;y; s) E Neocp(H), we then have 
— 2 = jj? 
one -¥es||; < Io — Dell + [ve -xesl||; 
2 2) 2 2492) 1,2 (11.21) 
< (6 -1)?r+ 6) p? = (8? +0?) pw’. 
Since ||xes — yell < 0u and h = ope — x ° s, using Lemma 11.10 it follows that 
ae lone —Xos\f2 
[E-r reas teal aa» 
E NE 21 — 0)? 4 


Defining vt ê v(1) = x ost — u*e and using (11.17) with a = 1, (11.22), (11.21), (11.19), 
and (11.20), we get 


Itl = [Ax Ast. 


lA 


[avez 
lone — Xesl|; 
2(1 — 6)? 
(82 + 0y 
20 — oy 


0(1- =) w= on". 


pl*eAsll, 


lA 


Consequently, 
|=" ost — well < Out. (11.23) 


By using the right-hand side inequality in (11.18), and using (11.21) and (11.19), we have 


ja je Hone e VEO. a it Nea 
á Q- 8)u (1 —-@) vr 


where the strict inequality follows from 8 < ; and0<1-6/ Vr <1. 


One can easily see that || Ax o x! ll < 1 implies that e + Axex ' > 0, and therefore 


x =Ax+x= (e+ ixx) ox >Q. 


Note that, from (11.23), we have Amin o s+) > (1 — 0)u* > 0, and therefore x ost > 0. 
Since x" > 0 and x" and s* operator commute, we conclude that s+ > 0. Using the first 
equation of System (11.15), we get 


—+ — rE e Ay x # 
AX =A(¥+ Ax) = Ax +AAx=b, and hence X € Fyegcp- 
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By using the second equation of System (11.15), we get 
Alyt +5" = Alyy + Ay) + (5+ 4s) = Aly +54 ATAy+ As =e, 


and hence (yt;5") E P isoci: 

Thus, in view of (11.23), we deduce that (¥*; y+; 5°) € Nocp(“). Item (b) is there- 
fore established. Item (c) follows from item (b) and (Schmieta and Alizadeh, 2003, 
Proposition 29). The proof is now complete. E 


We finally have the following corollary, which states that Algorithm 11.1 can find an 
iterate meeting a given tolerance within a limited number of iterations, determined by a 
logarithmic function. 


Corollary 11.2 Let @ and 6 as given in Theorem 11.4 and (x°; y°; s) E€ Neocp(u). Then 
Algorithm 11.1 generates a sequence of points {(x; y; s)} C Msocplu) such that 


k 
T ô T 
x s” = (: — z) x® s®, Yk>0. 
r 


Moreover, given a tolerance e > 0, Algorithm 11.1 computes an iterate {(x; y®; s®)} 
satisfying x" s < ein at most 


xO sO 
a E 


iterations. 


Proof: Looking recursively at item (a) of Theorem 11.4, for each k we have that 


k 
ô 
x6" = ( - +) x's < e. 
r 


By taking natural logarithm of both sides, we get 


tial ies] bein |. 
vr xO g(0) 


which holds only if the inequality k(—6/,/r) < In(e/x's) holds, or equivalently, 
k > 6-4/7 In(x"s /e). The proof is complete. E 


11.6 A Homogeneous Self-Dual Algorithm 


In this section, we present a homogeneous self-dual algorithm for SOCP. The material of 
this section has appeared in (Alzalg, 2014a, Section 3). The algorithm presented in this 
section for SOCP generalizes the one proposed in Section 10.7 for linear programming. 


11.6 A Homogeneous Self-Dual Algorithm 


The following primal-dual SOCP model provides sufficient conditions (but not always 
necessary) for an optimal solution of (PISOCP) and (DISOCP). 


Ax = b, 
Aly+s=c, 
x's = 0, 
x,s > 0. 


The homogeneous SOCP model for the pair (PISOCP) and (DISOCP) is as follows: 
Ax —br = 0, 
—Aly -s +er 
—c'x +b'y -K 
x 


(11.24) 


ll 


0 
0 
0, (11.25) 
0 
0 
0 


a 
IV IV IY IY 


The first two equations in (11.25), when r = 1, establish conditions for primal and dual 
feasibility (with x, s > 0) and exhibit reversed weak duality. As a result, these equations, 
when combined with the third equation after setting x = 0, define optimal solutions for 
both the primal and dual problems. It is worth noting that when we homogenize rt (i.e., 
make it a variable), introduce the necessary dual variable into the third equation, incorpo- 
rate the artificial variable x to ensure feasibility, and include the third equation from (11.25), 
we achieve self-duality. 

It can be easily shown that x's + tx = 0. The following theorem relates (11.24) to (11.25), 
and it is easily proved. 


Theorem 11.5 The primal-dual SOCP model (11.24) has a solution if and only if the 
homogeneous SOCP model (11.25) has a solution 


*;s*;r*;*) € E}, xR” xE}, XR, XR}, 


(x*;y 


such that t* > 0 and x* = 


At each iteration of the homogeneous interior-point algorithm designed for solving 
(PISOCP) and (DISOCP), a key step involves computing the search direction (Ax; Ay; As). 
This direction is determined from the symmetrized Newton equations with respect to an 
invertible vector p, which is selected as a function of (x; y; s). These equations are defined 
by the system: 


A Ax —b At = nF, 
— A’ Ay —As +e At = nq, 
—c’ Ax + b! Ay - Ac =n, (11.26) 
KAT + TAK = YH- TK, 
(Q,Ax) © (Qas) + (Qx) © (QpiAs) = yue — (Qx) ° (Qpa), 
where 

F, £ þr — Ax, r, = Aly +s- tc, 
Tg Sclx—b'y+x, u ê a's +16), 


and y and y are two parameters. 


387 


388 


11 Second-Order Cone Programming 


The vectors x and s do not necessarily operator commute. However, as we discussed in 
Section 11.5, it is often advantageous to analyze interior-point methods under the assump- 
tion that x and s operator commute. To facilitate this analysis, we will now introduce a 
scaling transformation to the primal-dual pair (PISOCP) and (DISOCP). This scaling will 
ensure that the scaled decision variables in the resulting pair of problems, which should 
remain equivalent to the original (PISOCP) and (DISOCP), indeed operator commute. To 
achieve this, we will employ an effective scaling method demonstrated in (11.14). With this 
change of variables, the transformed pair (PISOCP) and (DISOCP) becomes: 


min c max b'y 
(PISOCP) st. Ax=b, (DISOCP) st. A'y+s=c, 
x > 0; s>0. 


Note that x's = x's (see (11.15)). From (Alizadeh and Goldfarb, 2003, Lemma 31) it 
can be seen that the search direction (Ax; Ay; As) solves System (11.26) if and only if 
(Ax; Ay; As) solves the system: 


A Ax —b At = nP, 
— A' Ay —As +c At = nha, 
=c" Ax + b' Ay —- Ak =nf, (11.27) 
K At + T Ak = yy -TK, 
Ax os + x o As = yye-—Xes, 
where 

f, = br — Ax, fA ne 

pA 2 


c'x —b'y+k, H 1a S+TK). 


E 2r+1 


For each choice of p, we get a different search direction. The three choices of p that we 
discussed in Section 11.5 are the most common in practice. We emphasize that (Ax; Ay; As) 
is the result of applying Newton’s method to the primal and dual feasibility and complemen- 
tarity relations arising from the scaled problems (PISOCP) and (DISOCP). It depends on the 
choice of p, while (Ax; Ay; As) results as a special case when p = e. 

We state the generic homogeneous algorithm for solving the pair (PISOCP) and (DISOCP) 
in Algorithm 11.2. 

The following theorem, which is known to hold, gives the computational complexity 
(worst behavior) of Algorithm 11.2 in terms of the rank of the underlying second-order cone. 


Theorem 11.6 Let e) > 0 be the residual error at a starting point, and e > 0 be a given 
tolerance. Under Assumptions 11.2 and 11.1, if the pair (PISOCP) and (DISOCP) has a solu- 
tion (x*; y*; s*), then Algorithm 11.2 finds an e-approximate solution (i.e., a solution with 
residual error less than or equal to e) in at most 


Pa ii ripe (2) 


iterations. 


Exercises 


We point out that the result in Theorem 11.6 for SOCP is the counterpart of that in 
Theorem 10.12 for linear programming. 


Algorithm 11.2: Generic homogeneous self-dual algorithm for SOCP 
Input: Data in Problems (PISOCP) and (DISOCP), (x;y; $; 7; K) £ (e;0;e;1;1) 
Output: An approximate optimal solution to Problem (PISOCP) 

1: choose a scaling element p and compute (x, s) 

2: while a stopping criterion is not satisfied do 

3: | choose y,y 

4: | compute the solution (Ax; Ay; As; At; Ax) of the linear system (11.27) 
s5: | compute (Ax; As)by applying inverse scaling to (Ax; As) 
6: | compute a step length 6 so that 

x+0Ax>0 

s+0As > 0 

t+0At>0 

k+0Ak >0 

7: | set the new iterate according to 

(x: y; S; T; K) Ê sy; 8; T; K) + O(Ax; Ay; As; AT; Ax) 

8: end 


In summary, SOCP stands as a powerful mathematical framework, adept at solving a 
wide array of optimization problems, particularly those involving complex constraints and 
nonlinear objectives, offering an efficient tool in diverse fields such as engineering, finance, 
and machine learning. 

Exercises 


11.1 Prove Lemma 11.2. 


11.2 The pth-order cone of dimension n is defined as 


pr È {x € E" : xX > Illl} 


where ||:|| is the p-norm of é: 


p21 


m 1/p 
léllp := (Zir) , for eR”. 
i=1 


When p = 2, the pth-order cone reduces to the second-order cone, that is, E} = Po: 
In the proof of Lemma 11.3, we used the Cauchy-Schwartz inequality to show that 
the second-order cone is self-dual (i.e., px = P3). More generally, use the Hdlder’s 
inequality to show that Pir = P} for p €E [1, œ], where q is the conjugate of p. 


11.3 The nth-dimensional elliptic cone is defined as 


Ki, = {x= ERXR T! : x > |M}, (11.28) 
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11.4 


11.5 


11.6 


11.7 


11.8 


where M be a non-singular matrix of order n—1. Clearly, when M =I,,_,, the 
elliptic cone reduces to the second-order cone, that is, E” = Les E In the proof of 
Lemma 11.3, we showed that the second-order cone is self-dual (i.e., (C7 Pa 


= K7, > More generally, show that 


*— 
ea es 


Prove item (c) in Lemma 11.8. 
Prove Lemma 11.9. 
Prove Lemma 11.10. 


Implement Algorithm 11.1 and test it on the class of instances of the pair (PISOCP) 
and (DISOCP) with n = 2m and 


c ê 10e-21+4 rand (n,1) € E”, 
b 4 10e-21+4 rand (m,1) € R”, 
A £ |Â : Randn (m,n - m)| €R™", 


where, for 1 < i,j < m, 


2, ifi=j-1, 

100, if i=j, 

j -2, if i=j+1, 
0, otherwise. 


> 
l> 


Here, 1 is a vector of ones with with an appropriate dimension, and rand(-, 1) 
(respectively, Randn(-,-)) is a random vector (respectively, matrix) with the indi- 
cated dimension (respectively, size). Take x = e € E” and yo = 0 € R” as your 
initial strictly feasible points. You may also take e = 107%, ø = 0.1, and p = 0.99. 


In this and the following exercises, we practice writing the homogeneous model of 
an SOCP problem and the corresponding search direction system. The underlying 
problems are SOCPs, in primal and dual standard forms, with block diagonal struc- 
tures. Let r > 1 be an integer. For i=1,2,...,r and k = 0,1, ... , K, let m, ng, ny; be 
positive integers such that n; = Dy, ni 


Let the data 
Wo = (Wos Woz» --- , Wor), Where Wp; € Rono; 
Co Ê (Coi; Co2; --- 3 Cor), Where Cy; € E”o; 
B, = (By, By, --- By), where By € R™*"«; 
W, = (Wig, Win, --- , Wg), where Wp; E Rx; 
Ê (Chi; Cy; --- 3 Cpr), Where Cy € E”, 


Notes and Sources 


for k = 1,2, ... , K be given. Consider the SOCP problem: 


min c}xọ + clx; +-+- + LX 
s.t. WX, = h, 
BX) + Wx = h, 
r 0 1 i 1 (11.29) 
BgX + WrXx = hg, 
Xo X], Meas Xg = 0, 
where 
A 7 : : Noi» 
Xo = (Xo1; X02; --- >Xor), Where Xo; € E”o; 
A 7 š š Ehki 
Xy = (X1 Xk2; --- 3Xpkr) where xy; € E™, 


for k = 1,2, ...,K, are the primal decision variables, and họ E€ R™, h, E R™, k = 
1,2, ... , K, are right-hand side vectors. The dual of (11.29) is the problem: 


max hyo + hly; +- + hiyx 
s.t. Wyo + Bly, +- + BlYk +8 = Co 


M PaE (11.30) 
Wir + Sk = Cx, 
So; Sj» arts Sx = 0, 
where 
Y Ê Vos 3Vg) E REM, 
Sk Ê (Si Sk2i --- 5 Skr), Where s,, € E™, 


for k = 0,1, ... ,K, are the dual decision variables. Write the homogeneous model for 
the pair (11.29) and (11.30). 


11.9 Write the search direction system corresponding to the homogeneous model 
obtained in Exercise 11.8. 


Notes and Sources 


The roots of conic optimization can be traced to the work of mathematicians like Michael 
Todd, Yinyu Ye, and Stephen Boyd in the 1980s and 1990s (refer to Nesterov and Todd 
(1998); Boyd et al. (2004)). In particular, SOCP is a relatively recent development in the field 
of mathematical optimization, with its origins in the late 20th century. The foundations of 
SOCP can be traced back to the early 1990s, with significant contributions from researchers 
such as Yurii Nesterov and Arkadi Nemirovski (refer to Nesterov and Nemirovskii (1994)). 
SOCP was introduced as an extension of linear and quadratic programming, offering a more 
versatile framework to solve complex optimization problems. It has found applications in 
various fields, including engineering, finance, and machine learning, due to its ability to 
handle a wide range of convex optimization problems efficiently. 

In this chapter, we explored SOCP problems and their associated interior-point methods, 
providing a comprehensive understanding of this important class of optimization problems. 
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After studying the algebraic structure of the second-order cone, we delved into the the- 
ory and applications of SOCP. Additionally, we illuminated the interior-point methods that 
were developed for solving SOCP problems efficiently. 

As we conclude this chapter, it is worth noting that the cited references and others, such 
as Ben-Tal and Nemirovski (2001); Renegar (2001); Davidsson (2013); Aliprantis and Tourky 
(2007); Jacobson (1968), and Koecher et al. (1999), also serve as valuable sources of infor- 
mation pertaining to the subject matter covered in this chapter. The code that created the 
picture of the second-order cone in Section 11.2 was taken from the source file of Alzalg 
and Alioui (2022). Exercise 11.2 is due to Alzalg (2011a). Exercise 11.3 is due to Alzalg and 
Pirhaji (2017b). Exercise 11.7 is due to Alzalg (2018). Exercises 11.8 and 11.9 are due to 
Alzalg (2014a). 
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In Chapters 10 and 11, we have studied linear programming and second-order cone pro- 
gramming, respectively. In modern convex optimization, the class of optimization that is 
an immediate enlargement of second-order cone optimization is the so-called semidefinite 
optimization. Figure 12.1 shows graphical relationships among different classes of opti- 
mization problems. Semidefinite programming (SDP for short) problems, which include 
linear programming problems and second-order cone programming problems as special 
cases, are a class of convex optimization problems in which the variable is not a vector that 
is required to be nonnegative, but rather a symmetric matrix that is required to be positive 
semidefinite (see Definition 3.3). This chapter delves into the SDP problems. We also refer 
to Todd (2001) for an excellent survey paper on this topic. 


12.1 The Cone of Positive Semidefinite Matrices 


Recall that a square matrix is positive semidefinite (respectively, positive definite) if it is 
symmetric and all its eigenvalues are nonnegative (respectively, positive). As an alternative 
to the above definition, a matrix U € R’™" is positive semidefinite (respectively, positive 
definite) if it is symmetric and x'Ux > 0 for all x € R” (respectively, x'Ux > 0 for all 
x € R” — {0}). An immediate corollary here is that xx! is a positive semidefinite matrix 
for all x € R”. 

This section aims to introduce tools needed to study the SDP problems. We start this 
by introducing notations that will be used throughout this chapter. We use S” to denote 
the space of real symmetric n x n matrices. An identity matrix of appropriate dimension is 
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Semidefinite programming 


Second-order cone programming 


Convex quadratic programming 


Linear programming 


Figure 12.1 A Venn diagram of different classes of optimization problems. 
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Figure 12.2 A 3D plot of the boundary of the cone of the positive semidefinite matrices in S?. 
The boundary is the continuous surface shaped by the above blue mesh. 


denoted by I. The set of the positive semidefinite matrices in S” is a convex self-dual cone 
(see Lemma 12.1). See Figure 12.2 which shows a 3D plot of the boundary of the cone of 
the 2 x 2 positive semidefinite matrices. 

For U,V € S”, we write U > 0 (U > 0) to mean that U is positive semidefinite (positive 
definite), and we use U > V or V x U to mean that U — V 20. 

The bilinear map o : R™” x R™" — R®” is defined as 


UoV ê SUV + vU). (12.1) 
The Frobenius inner product e : R” x R™™" — R is defined as 


U e V Ê trace (UTV). 
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It is known that the space S” under the bilinear map o : S” x S” ——> S” defined in (12.1) 
forms a Euclidean Jordan algebra (see Faraut (1994); Schmieta and Alizadeh (2003) for 
definitions) equipped with the standard inner product (U,V) = Ue V = trace (U o V) = 
trace (UV). The Frobenius norm of a matrix U € R””" is defined as 


lUll- ê U $° U = PAO), 
i 


where 4;s(U) are the eigenvalues of the matrix U. 
It is known that, for any U, V € S”, we have (see Horn and Johnson (1990)) 
IU + Vile = lle + IVI} + 2U e V, 
UV; < IUl Wile < IIUl WV Ile. (12.2) 
UV; < lUll IVl < WU lle WV Ile. 


where ||U||, = max {||Ux||, : ||x||, = 1} = max;,|A,(U)| is the operator norm (or 2-norm) of 
the matrix U. 

The following properties of matrices are taken from (Todd, 2001, Section 3) and are given 
here without proofs. For proofs, see, for example, Horn and Johnson (1990); Watkins (1991) 
and Todd (2001). 


Property 12.1 (Trace commutativity) If A € R”*" and B e R™"”, then trace(AB) = 
trace(BA). 


In particular, like any inner product, we have that Ue V = V e U for U,V e R”™*". The 
following property in S” is the counterpart of Property 11.1 in E”. 


T 


Property 12.2 (Spectral decomposition in S”) Any A € S” can be expressed in 
exactly one way as a product A = Q(A)A(A)Q(A)', where Q(A) € R™" is an orthogonal 
matrix whose columns are the eigenvectors of A, and A(A) € R™" is a diagonal matrix 
whose diagonal entries are the eigenvalues of A. 


Recall that a square matrix is called lower triangular if all the entries above the main 
diagonal are zero. A unit lower triangular matrix A is a lower triangular matrix with a; = 1 
for 1 <i < n. Therefore, a unit lower triangular matrix has the form: 


1 : 
x 1 

x xX I 
Unit lower triangular matrices arise in the following property about the LDL' 
decomposition, which is a variant of the spectral decomposition. 


Property 12.3 (LDL' decomposition) Any A > 0(A > 0) can be expressed in exactly 
one way as a product A = L(A)D(A)L'(A), where L(A) € R™" is a unit lower triangular 
matrix, and D(A) € R” is a diagonal matrix whose diagonal entries are nonnegative 
(positive). 
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An immediate corollary of Property 12.3 is that every A > 0 (A > 0) has a square root 
A12 > 0 (A12 > 0). Simply, we take A! = L(A)D'/?(A)L'(A). Another immediate corol- 
lary of this property is that every A > 0 has an inverse. Simply, we take A~! = L(A)D71(A) 
L'(A). The Cholesky decomposition is a variant of the LDL’ decomposition. 


Property 12.4 (Cholesky decomposition) Any A > 0 can be expressed in exactly one 
way as a product A = LL", where L € R™" is lower triangular and has all main diagonal 
entries positive. 


The following property is used to represent the quadratic forms with the Frobenius inner 
product. 


Property 12.5 (Representing quadratics) If A € S” andx € R”, then 


x'Ax =Aexx'. 


In the following property and in what follows, we say that two matrices are simultane- 
ously diagonalized if they have spectral decompositions with the same Q. 


Property 12.6 (Commutativity and symmetry) IfA, B € S”, then A and B commute 
if and only if AB is symmetric, if and only if A and B are simultaneously diagonalized. 


The Schur complement property presented in Property 12.7 has many applications 
in SDP. 


Property 12.7 (Schur complement) Let A and B be symmetric matrices and A > 0. 
Then 
T 
= p >0(>0) > C-B'A”B > 0 (> 0). 
B C 
The matrix C — B'A7!B is called the Schur complement of the submatrix A. 


Associated with U, € S"(i = 1,2, ... , m), we define the linear operator V : S” > R” and 
its adjoint operator V’* : R” > S” as 


U, eX 
U,eX < 
Ux 4} °?°" |eR", ad u*z4 zu, es”, (12.3) 
s l=1 
Un 0X 


respectively. Note that 


(VX)'z= YW, eXz = (Seu, ° x) =U*zeX, (12.4) 
l=1 


l=l 


for X € S” and z € R”. 
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Associated with each nonsingular matrix P € R””, the symmetrization operator Hp : 
R™" ——-> S" is defined as 
H pM) ê (PMP + (pMP~)') (12.5) 
for M e R”. We will use the operator #>(-) in order to symmetrize the optimality 
condition system, which is needed for applying Newton’s method. 
To end the notations, we point out that in the sequel Diag(x) € S” is written for the 
diagonal matrix with the vector x € R” on its diagonal. 


12.2 Semidefinite Programming Formulation 


Within this section, we aim to familiarize readers with the SDP problem, providing an 
understanding of its form. Furthermore, we endeavor to delve deeper into this domain 
by demonstrating how various established categories of optimization problems can 
be approached through the lens of SDPs. This includes showcasing how certain well- 
recognized classes of optimization problems find their representation and resolution 
within the realm of SDP. 


12.2.1 Problem Formulation 


We will be concentrating on SDP problems in primal standard form and in the correspond- 
ing dual form. The problem in primal form can be written as 


min Cex 
(PISDP) s.t. Aje X = b, i=1,...,m, 
X = 0, 
where the data C, A;,i=1,...,m, are real symmetric n x n matrices while b is a real m- 


vector and the variable X is a real symmetric n x n matrix. 
It is convenient to introduce a slack matrix S and rewrite the problem as 


max b'y 
(DISDP) s.t. Yi, yA; + S = C, 
S20. 


Here the variable y is a real m-vector while S is a real symmetric n X n matrix. 
Using the notations introduced in (12.3), we can write our problems more compactly as 


min C èe X max b'y 
(PISDP) s.t. AX =b, (DISDP) st. A*y+S=C, 
X > 0; S 2 0. 


12.2.2 Formulating Problems as SDPs 


In this part, we formulate four general classes of optimization problems as SDPs. We start 
with linear optimization. 
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12.2.2.1 Linear Programming 
The linearity in linear programming can be transformed into a quadratic form that fits the 
criteria for SDP. Through this transformation, a linear program can be viewed as a specific 
instance of SDP, underscoring the relationship and demonstrating how the latter encap- 
sulates a broader spectrum of optimization problems, including the more restrictive linear 
programming. 

It is easy to see that the following pair of primal and dual linear programming problems: 


min c'x max b'y 
st. Ax=b, st. Aly+s=c, 
x > 0; s > 0, 


is equivalent to the pair (PISDP) and (DISDP), respectively, with 
C = Diag(c), X = Diag(x), S = Diag(s), and A; = Diag(a,), 


where aq, is the ith row of A. 


12.2.2.2 Convex Quadratic Programming 
In convex quadratic optimization problems, we minimize a strictly convex quadratic func- 
tion subject to affine constraint functions: 


min q(x) £x'Qx+e'x 
s.t. Ax = b, (12.6) 
x> 0. 
Since Problem (12.6) is strictly convex, the matrix Q must be a symmetric positive defi- 
nite matrix (i.e., Q = Q7 and Q > O). Let Q = LL' (see Property 12.4). Then, using Schur 
complements (Property 12.7), the inequality q(x) < t can be written as 


I L'x SÖ 
x'L t—c'x] ~ 


Therefore, the quadratic optimization problem (12.6) can be formulated as the following 
SDP relaxation. 


min t 
s.t. Ax = b, 
I L'x 
20, 
an t— e 
x>0. 


We also point out that, more generally, convex quadratically constrained quadratic 
optimization problems can also be formulated as SDP problems (see, e.g., (Todd, 2001, 
Example 5)). 


12.2.2.3 Second-Order Cone Programming 
Consider the second-order cone programming problem (as introduced in Section 11.2): 
min c'x 
s.t. Ax = b, 
x20. 


12.3 Applications in Combinatorial Optimization 


The second-order cone constraint x > 0 is the inequality x, > ||x|| (ie., x € E%; see 
Definition 11.1). Using Schur complements (Property 12.7), we have that 


x= (x) 20 Ki *| > 0, (12.7) 
0 


that is, Arw(x) 20, where Arw(x) is the arrow-shaped matrix introduced in 
Definition 11.4. 


12.2.2.4 Rotated Quadratic Cone Programming 

Rotated quadratic cone programs were introduced in Section 11.2 in the context of second- 
order cone optimization. Let n be a positive integer and M be a nonsingular matrix of order 
n — 2. Recall that the nth-dimensional rotated quadratic cone is 


K” = {x = (3x58) ERX R X R"? : xox, > IIRI’, x) = 0.x, > 0}. 
Recall also that the constraint on x that satisfies the inequality 2x,x, > ||X||? is called the 
hyperbolic constraint. As mentioned in Section 11.2, in a rotated quadratic cone optimiza- 


tion problem, a linear objective function is minimized subject to linear constraints and 
hyperbolic constraints. Note that 


(tae) EK” = (2x9 +X; 2X0 — X; 2k) € E? (By (11.6)) 
(2X) +x) (2x9 — x1; 2%) 
(2X xX; 28) 2x +X 


€ S". (By (12.7) 


This means that a rotated quadratic cone optimization problem can be expressed as an SDP 
problem because the hyperbolic constraint is equivalent to a linear matrix inequality. 


12.3 Applications in Combinatorial Optimization 


In this section, we describe three combinatorial applications of SDP. For non-combinatorial 
applications, we point out that the applications that we described in Section 11.3 in the 
context of second-order cone programming were first formulated as applications of SDP 
(see Vandenberghe and Boyd (1999)). The focus in this chapter is only on those applications 
in combinatorial optimization. The material of this section has appeared in some sections of 
Goemans (1998); Vandenberghe and Boyd (1999), and Todd (2001). For more applications 
of SDP, we refer the reader to Goemans (1998); Karger et al. (1998); Vandenberghe and Boyd 
(1999); Rendl (1999); Todd (2001), and Lidicky and Pfender (2021). In particular, the work 
of Lidicky and Pfender (2021) presents an interesting application in Ramsey theory. 


12.3.1 Shannon Capacity of Graphs 

In this problem, the objective is to bound the Shannon capacity, or the independence num- 
ber of graphs. We need some definitions. 

Definition 12.1 Let G = (V, E) be an undirected graph. Then: 


(a) A clique set (or simply, a clique) of G is a set S C V of mutually adjacent vertices. 
(b) A clique cover of G is a collection C of cliques that together include all V. That is, 
V C Use S. 
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(c) The clique cover number of G, denoted by 7(G), is the minimum cardinality of a clique 
cover of G. That is, 7(G) is the smallest number of cliques of G whose union covers the 
set V. 


Note that the clique cover number of a graph and the chromatic number (see 
Definition 4.7) of its complement are identical, that is, Z(G) = 7(G). 


Definition 12.2 Let G = (V, E) be an undirected graph. Then: 


(a) An independent (or stable) set of G is a set S C V of mutually nonadjacent vertices. 

(b) The independence (or stability) number of G, denoted by a(G), is the maximum cardi- 
nality of an independent set G. In other words, a(G) is the largest number of vertices 
that can be colored with the same color in a graph coloring of G. 


Note that each node in a stable set must be in a different clique in a clique cover, hence 
we have 


a(G) < x(G). (12.8) 


If the equality holds in (12.8), then we say that G is a perfect graph. 

Computing both a(G) and Z(G) is a well-known NP-hard problem. In the context of the 
Shannon capacity problem, the goal is to approximate a function that falls between these 
two values. This intermediary function is referred to as Lovász’s theta function, denoted as 
0(G), and it represents the optimal solution to the SDP problem: 


max 11’ eX 


st. JTexX=1, (12.9) 
x = 0, for (i,j) € E, ` 
X =0, 


where 1 is a vector of ones with an appropriate dimension. 
Note that the SDP problem (12.9) is in primal form, but in maximization form. The dual 
of Problem (12.9) is the SDP problem (Todd, 2001): 


min s 
s.t. sI+ £ y;Mj>*1 1', 
Gj)EE 
where M; is the symmetric matrix that is all zero except for ones in the ijth and jith positions. 
Clearly, the above SDP models calculate the numbers a(G) and y(G) exactly if G is perfect. 


12.3.2 Max-Cut of Graphs 


In a max-cut graph problem, the objective is to find the cut of maximum weight in undi- 
rected graphs. This problem arises in finding the ground state of a spin glass; see Poljak and 
Tuza (1993). We need the following definition. 


12.3 Applications in Combinatorial Optimization 


Definition 12.3 Let G = (V, E) be an undirected graph with a nonnegative weight vector 
w = (Wy)i jer E RI, The cut determined by a subset S C V is the set 6(S) ê {(i,j) CE: 
i € S,j € S}, and its weight is defined as w(6(S)) ê Papes Wy 


In light of Definition 12.3, we can assume that the underlying graph is complete by letting 
wy =0 for all (i,j) € E (we let also w; = 0 for all i). To represent the cut 6(S), we introduce 
the variable x € R'!, which is defined as 

xadh ifie S, 
i \-1, ifieV-S. 
It follows that 
fasion 2, if (i,j) € 6(S), 
"I \0, if ij) € 6(S). 
We also define the matrix C € S|! as 
Cy = ee Ve Fj, 
LjMy/4, ifi=j. 
Note that 
w(ô(S)) 


ll 
S 


(ij)€6(S) 


= x'Cx=Cexx'. 


Because every (+1, —1)-vector in R!”! corresponds to a cut in the graph G = (V, E), the 
max-cut problem can be formulated as the integer program: 


max Cexx! 


12.10 
st x © {41,-1}/1, ( ) 
or equivalently, as the quadratic program: 
max x'Cx 
(12.11) 


s.t. x? =1,iEV. 


Observe that Problem (12.11) is linear in the products XiX, and these are the entries of the 
rank one matrix X xx". Observe also that X € s”, with x; = 1 for all i € V. It follows 


that Problem (12.10) can be written as 


max CexX 

to y= LieV 

an (12.12) 
X>0, 


X is of rank one. 
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Relaxing the last constraint in Problem (12.12), we obtain the SDP problem (Todd, 2001): 


max Cex 
St. x, =LiEV, 
X20. 


We also mention that Todd (2001) discusses two other different ways to arrive at an SDP 
relaxation of the max-cut problem. 


12.3.3 Combinatorial Topology Optimization 


The Ben-Tal and Bendsge’s paper (Ben-Tal and Bendsge, 1993) addresses a problem from 
combinatorial topology optimization of truss structures. Specifically, the authors investi- 
gate a scenario where a structure comprises k linear elastic bars connecting a set of p nodes. 
In this analysis, it is assumed that the geometry, including the topology and lengths of the 
bars, as well as the material, remain fixed. The primary objective of this study is to deter- 
mine the optimal sizes of the bars, which involves selecting appropriate cross-sectional 
areas for them. 

For i=1,2,...,k, and j=1,2,...,p, we define the following decision variables and 
parameters: 


ef * the external force applied on the jth node, 

e d; * the (small) displacement of the jth node resulting from the load force Íp 

o X Ê the cross-sectional area of the ith bar, 

eX, Ê the lower bound on the cross-sectional area of the ith bar, 

e X, = the upper bound on the cross-sectional area of the ith bar, 

e l; ê the length of the ith bar, 

e v =the maximum allowed volume of the bars of the structure, 

e Gx) ê Di x,G; is the stiffness matrix, where the matrices G; € S, i = 1,2, ... ,k, depend 
only on fixed parameters (such as length of bars and material). 


In the simplest version of the problem, one considers one fixed set of externally applied 
nodal forces Í j= 1,2, ...,p. Given this, the elastic stored energy within the structure is 
given by 


e=f'd, 


which serves as an indicator of the measure of the inverse of the structure’s stiffness. In 
view of the definition of the stiffness matrix G(x), we can also conclude that the following 
linear relationship between f and d: 


f=% d. 


The objective is to find the stiffest truss by minimizing e subject to the inequality I' x < v 
as a constraint on the total volume (or equivalently, weight) and the constraint x < x <x 
as upper and lower bounds on the cross-sectional areas. 

For simplicity, we assume that x > 0 and G(x) > 0, for all x > 0. In this case, we can 
express the elastic stored energy in terms of the inverse of the stiffness matrix and the exter- 
nal applied nodal force as follows: 


e=f'Gx) lf. 


12.4 Duality in Semidefinite Programming 


In summary, they consider the problem (see also Vandenberghe and Boyd (1999)): 
min f' G(x) 'f 
s.t. X<xX <x, 
Ix <v, 


which is equivalent to: 


min s 

s.t. f' Gx) <s, 
f f (12.13) 
X<xX <x, 
Ux<v. 


The first inequality constraint in (12.13) is just fractional quadratic function inequality con- 
straint and it can be formulated as a positive semidefinite constraint. This problem can be 
cast as an SDP problem as follows: 


12.4 Duality in Semidefinite Programming 


In this section, we offer a brief overview of the duality theory pertaining to SDP and intro- 
duce the complementarity condition as one of the optimality criteria for SDP. While much 
of the duality theory for SDP shares similarities with that of linear programming (as well as 
second-order cone programming), it is important to note that there are certain distinctions. 

Recall that a regular cone is self-dual if it equals its dual cone (see Definition 3.22). 
Now we need the following lemma. 


Lemma 12.1 The symmetric positive semidefinite cone S} is self-dual under the 
Frobenius inner product. 


Proof: We verify that the symmetric positive semidefinite cone S% equals its dual cone, 
which is defined as 
(St)* 4 {Xe S": XeY>0 forall Yest}. 


We first prove that (S”)* C S”. Assume that X ¢ S”, then there exists a nonzero vector 
y E R” such that X e yy! = y'Xy < 0, which shows that X ¢ (S")*. 
Now, we prove that S} C (S7)*. Letting X € S", then for any Y € S}, we have that 


X e Y = trace(XY) = trace (xeYx2) >0, 
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where we used the fact that X?YX? € S” to obtain the inequality. In fact, if U € S} and 
Qand A are respectively its corresponding orthogonal and diagonal matrices obtained from 
the spectral decomposition (see Property 12.2), then we have 


trace(U) = trace(QAQ') = trace(AQQ') = trace(A) = Dy > 0. 
i=1 


We have shown that X € (S")* whenever X € S". The proof is complete. a 


The weak duality property in SDP is encapsulated by Theorem 12.1, which asserts that 
the objective function values of primal feasible solutions consistently surpass those of dual 
feasible solutions. 


Theorem 12.1 (Weak duality in SDP) Let X and (y,S) be feasible for (PISDP) and 
(DISDP), respectively, then 


CeX-—b'y=SeX>0. 


Proof: Using (12.4), we have 


CeX—b'y = (A*y + S) è X —(AX)'y 

= A*yeX+SeX-(AX)'y 
A*yeX+SeX-—A*yex 
= SeX>0, 


where the last inequality follows from the self-duality of the cone of the symmetric positive 
semidefinite matrices. The proof is complete. a 


As we mentioned in Chapter 11, it is important to note that the strong duality property 
may not hold in general conic optimization, as highlighted in Nesterov and Nemirovskii 
(1994). Nevertheless, even in cases where strong duality does not apply, we can establish a 
somewhat weaker property that consistently holds in conic optimization. In the context of 
SDP, we will now outline the conditions under which this slightly weaker property remains 
valid. We say that the primal problem is strictly feasible if there exists a primal feasible point 
Å such that X > 0. We make the following assumption for convenience. 


Assumption 12.1 Them matrices A,,A,,...,A,, are linearly independent in S”. 

Now we state and prove the following semi-strong duality result. 
Lemma 12.2 (Semi-strong duality in SDP) Consider the primal—dual pair (PISDP) 
and (DISDP). If the primal problem is strictly feasible and solvable, then the dual problem 


is solvable and their optimal values are equal. 


Proof: By the assumption of the lemma, the primal problem is strictly feasible and solv- 
able. So, let X be an optimal solution of the primal problem where we can apply the KKT 
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conditions (see Theorem 11.1). This implies that there are Lagrange multipliers y and S 
such that (X, y, S) satisfies: 


AX =b, 
A*y+S=C, 
XeS=0, 
X,S20. 


This means that (y, S) is feasible for the dual problem. Let (v, Z) be any feasible solution 
of the dual problem, then we have that b'v<CeX=SeX+ b'y = b'y, where we used 
the weak duality to obtain the inequality and the complementary slackness to obtain the 
last equality. Thus, (y, S) is an optimal solution of the dual problem and C eX = b'y as 
desired. a 


The following strong duality result can be obtained by applying the duality relations to 
our problem formulation (see also (Todd, 2001, Theorem 4.1)). 


Theorem 12.2 (Strong duality in SDP) Consider the primal—dual pair (PISDP) and 
(DISDP). If both the primal and dual problems have strictly feasible solutions, then they 
both have optimal solutions X* and (y*, S*), respectively, and 


p* £CeX* =d* £ b'y*(i.e.,X* e S* = 0). 


The following lemma describes the complementarity condition as one of the optimality 
conditions of SDP. This lemma is not hard to show using the spectral decomposition of X, 
and considering its positive and zero eigenvalues separately. 


Lemma 12.3 (Complementarity condition in SDP) IfX, S > 0, then 
XeS=0 — XS=0. 


As a result of Lemma 12.3, the complementarity slackness condition for the primal and 
dual SDP problems (PISDP) and (DISDP) can be equivalently represented by the equation 
XS = 0. From the above results, we get the following corollary. 


Corollary 12.1 (Optimality conditions in SDP) Consider the primal—dual pair 
(PISDP) and (DISDP). Assume that both the primal and dual problems are strictly feasible, 
then (X, (y,S)) E S” x R” x S” is a pair of optimal solutions to the SDP (PISDP) and 
(DISDP) if and only if 


AX =b, 
A*ty+S=C, 
XS = 0, 
X,S20. 


(12.14) 


We have established the duality relations in SDP. The focus in the remaining part of this 
chapter is to solve SDP algorithmically. 
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12.5 A Primal-Dual Path-Following Algorithm 


In Section 11.5, we presented a primal-dual path-following algorithm for solving SOCP 
problems. In this section, we present a primal—dual path-following algorithm for solving 
SDP problems. The material presented in this section is based on, and similar to, any 
primal-dual path-following algorithm proposed for SDP (see for instance Touil et al. 
(2017)). The general scheme of the path-following algorithms for SDP is as follows. We 
associate the perturbed problems to semidefinite programming problems (PISDP) and 
(DISDP), then we draw a path of the centers defined by the perturbed KKT optimality 
conditions. After that, Newton’s method is applied to treat the corresponding perturbed 
equations in order to obtain a descent search direction. 

Let u > 0 be a barrier parameter. The perturbed primal problem corresponding to the 
primal problem (PISDP) is 


min f,(X) =CeX-—yp In det(X)+nylnp 
(PISDP,) st. AX =b, 
X >0, 
and the perturbed dual problem corresponding to the dual problem (DISDP) is 


max g,(y,S) = b'y+y In det(S)—nylny 
(DISDP,) st. A*y+S=C, 
S>0. 


Now, we define the following feasibility sets: 


Faspp {X € S” : AX =b, XS 0}, 
Fospe = {0,S) E R” xS” : A*y+S=C, S>0}, 
F RISDP ê {X eS": AX =b, X>0}, 

Pepp £ {0,9 ER” XS": A*y+S=C, S>0}, 
o A o o 

spop =  pispp X Fpisor' 


We also make the following assumption about the primal-dual pair (PISDP) and (DISDP). 
Assumption 12.2 The set F pp is nonempty. 


Assumption 12.2 requires that Problem (PISDP u) and its dual (DISDP a) have strictly fea- 
sible solutions, which guarantees strong duality for the SDP problem. Note that the feasible 
region for Problems (PISDP,,) and (DISDP,) is described implicitly by Fpp- Due to the coer- 
civity of the function f, on the feasible set of (PISDP u)» Problem (PISDP P) has an optimal 
solution. 

The following lemma proves the convergence of the optimal solution of Problem (PISDP,) 


to the optimal solution of Problem (PISDP) when y approaches zero. 


Lemma 12.4 Let X , be an optimal primal solution of (PISDP,,), then X = lim at „isan 
optimal solution of Problem (PISDP). 
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Proof: Let f,,(X) 2 f(X, u) and f(X) Ê f(X, 0). Due to the coercivity of the function f, on the 
feasible set of (PISDP,,), Problem (PISDP „) has an optimal solution, say X w such that 


Vf D) = Vf X pu) = 0. 


Then, for all X € F gies we have that 


FX) > FX Mw) + K-X,) VSE pA + O- WFLX, Ww) 
> fX 4) + wln det X, —nulny-ny 
CeX,-nulndet X, +nyuln y+ windet X, -—nyln p -ny 


> 
> CeX,-ny. 


Since X was arbitrary in F, 


isis this implies that 


min f(X)>CeX,-—nyw>CeX, =f(X,). 


o 
XEF psp 


On the other side, we have f X W) 2 minger? f(X). As u goes to 0, it immediately follows 
PISDP 


that f(X) = min xer?_f(X). Thus, X is an optimal solution of Problem (PISDP). The proof 
PISDP 
is complete. a 


12.5.1 Newton’s Method and Commutative Directions 


As we mentioned, the objective function of Problem (PISDP D is strictly convex, hence the 
KKT conditions are necessary and sufficient to characterize an optimal solution of Problem 
(PISDP_,). Consequently, and in light of Lemma 12.3, the points X , and (y,,,S,) are optimal 
solutions of (PISDP,,) and (DISDP.,) respectively if and only if they satisfy the perturbed 
nonlinear system 


AX =b, X>O, 
A*ty+S=C, S >0, (12.15) 
XS = uI, pw > 0, 


where I is the identity matrix of S”. 

We call the set of all solutions of system (12.15), denoted by (X wow Sp) with „u > 0, the 
central path. We say that a point (X,y, S) is near to the central path if it belongs to the set 
Nspp(), which is defined as 


Nop) Ê {(Xy,S) © Frispp X Foispp : dso (X: S) < 64,0 € 0, )}, 
where 
dspp(X, S) 2 |X o S— HI lp. 


Since for X, S € S”, the product XS is generally not in S”, so the left-hand side of System 
(12.15) isa map from R?” x R” x S” to S” x R” x S”. Thus, System (12.15) is not a square 
system when X and S are restricted to S”, which is needed for applying Newton’s method. 
A remedy for this is to make System (12.15) square by modifying the left-hand side to a map 
from S” x R” x S” to itself. To this end, we use the symmetrization operator #p(-) defined 
in (12.5). The following lemma is due to Zhang (1998). 
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Lemma 12.5 For M € S”, nonsingular P € R”, and a scalar t, we have 


H (M)=tl <> M=]. 


In light of Lemma 12.5, for any nonsingular matrix P, System (12.15) is equivalent to 


AX =b, X>0O, 
A*ty+S=C, S >0, (12.16) 
H (XS) = wl, p > 0. 


Recall that #,(XS) = 5(PXSP™? + P-!SXP"). So, an alternative way to view the above 
development is to scale (PISDP) so that the variable X is replaced by X £ PXP" and to scale 
(DISDP) so that the variable S is replaced by S ê P7" SP7. 

The need for the above symmetrization occurs because X and S do not commute (see 
Property 12.6). So, we choose P so that the scaled matrices commute. Denote by C(X, S) the 
set of all matrices so that the scaled matrices commute. That is, 


C(X, S) ê {P e S” : P7! exists, and PXP" & P7" SP7! commute }. (12.17) 


Now, we can apply Newton’s method to System (12.16) and obtain the following linear 
system 


AAX = 0, 
Ax*Ay+AS = 0, (12.18) 
H (XAS + AXS) = oul — H(XS). 
where (AX, Ay, AS) E€ S” x R” x S” is the search direction, ø € (0,1) is the centering 
parameter, and p= ix eS= 1X e S is the normalized duality gap corresponding to 
(X, y, S). In fact, 


+X eS=-+(PXP")e (Fr eres lyes. 
n g n n 


Solving the scaled Newton system (12.18) yields the search direction (AX, Ay, AS). Note 
that the search direction (AX, Ay, AS) belongs to the so-called the MZ family of directions 
(due to Monteiro (1997); Zhang (1998)). In fact, such a way of scaling originally proposed for 
SDP by Monteiro (1997) and Zhang (1998), and after that it was used for second-order cone 
programming in Schmieta and Alizadeh’s paper (Schmieta and Alizadeh, 2003). Clearly, 
the set C(X, S) defined in (12.17) is a subclass of the MZ family of search directions. Our 
focus is in matrices P € C(X,S). We discuss the following three choices of P (see (Todd, 
2001, Section 6)): 


e The first one is to choose P = X~'/2, which gives X = I. 


e The second one is to choose P = S!/?, which gives S = I. 
e The third choice of P is given by P = (X'/2(X'/28X'/?)-V/2X1/2)-1/2, which yields 
P?XP2 = S, and therefore X = S. 


As we mentioned in Chapter 11 for the case of second-order cone programming, the 
first two choices of directions are respectively called the HRVW/KSH/M direction and 
dual HRVW/KSH/M direction (due to Helmberg et al. (1996); Monteiro (1997); Kojima 
et al. (1997)). The third choice of direction is called the NT direction (due to Nesterov and 
Todd (1998)). 
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12.5.2 Path-Following Algorithm 


We formally state the path-following algorithm for solving SDP problem in Algorithm 12.1. 
Algorithm 12.1: Path-following algorithm for SDP 
Input: Data in Problems (PISDP) and (DISDP), k = 0, (X,y©,S) E€ Mpp (u), 
e>0, 6,0 e (0,1) 
Output: An c-optimal solution to Problem (PISDP) 
1: while X e S® > edo 
2: | choose P® Ec (x, S®) 
3 | setp® ê 1 xX® es 
H® 4 oP yh] -XV o SV 
4: | compute (AX; Ay; AS®) by solving the scaled system (12.18) to get 
Ay® =-(A(s®* ex) A‘) A (S> oH) 
AS® = —A*Ay® 
AX® = SH. ( HH -XV-A s®) 
5: | set the new iterate according to 
XD £ XV 4 Qhaxw 
yD 2 yl 4 gay 
SUD A SO 4 gl ago 
6 | setk=k+1 
7: end 


Algorithm 12.1 selects a sequence of displacement steps {a} and centrality parameters 
{o} according to the following rule: For all k > 0, we take o = 1 —6/,/n, where ô € 
[0, yn). The paper Touil et al. (2017) discusses in Section 3 various selections for calculating 
the displacement step a. 

In the rest of this section, we prove that the complementary gap and the function f, 
decrease for a given displacement step. The proof of this result depends essentially on the 
following lemma. 


Lemma 12.6 Let (X,y, S) € int S% x R” x int S", (X,y, S) be obtained by applying scal- 
ing to (X, y, S), and (AX, Ay, AS) be a solution of System (12.18). Then we have 


(a) AX è AS=0. 
(b) Xe AS + AX eS = trace(H), where H ê onl — XS such that o € (0,1) and p= 1 
XeS 


(c) XteSt = (1-a(1-2)) X eS, Va € R, where X+ ê X + «AX and St Ê S + aAS. 


Proof: From the first two equations of System (12.18), we get 
AX @ AS = —AX è A* Ay = —(AAX)' Ay = 0. 


This proves item (a). 
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We prove item (b) by noting that 


trace(H) = trace (oul — X ° S) 
trace (X o AS + AX ° S) 
= trace (X e AS) + trace (AX °- S) =X eAS+AX eS, 


where we used the last equation of System (12.18) to obtain the first equality. Item (c) is left 
as an exercise for the reader (see Exercise 12.1). The proof is complete. a 


The result in the following theorem is essentially those in (Touil et al., 2017, Lemmas 4.2 
and 4.3). The result given here in SDP is the counterpart of that in Theorem 11.3 
for second-order cone programming and its proof is also the same as the proof of its 
counterpart. 


Theorem 12.3 Let (X,y,5S) and (x E y+, S+) be strictly feasible solutions of the pair of 
problems (PISDP,,) and (DISDP,) with 

(X+, y+, S+) = (X + «AX, y+ adAy, S+aAS), 
where a is a displacement step and (AX, Ay, AS) is the Newton direction. Then 


(a) XteSt<XeS. (b) f,(X*) <f,(X). 


Proof: Note that 
XteSt= (1-a(1-$)) XeS<Xes, 


where the equality follows from item (d) of Lemma 12.6 and the strict inequality follows 
from (1 — a(1 — o /2)) < 1 (as æ > 0 and o € (0, 1)). This proves item (a). 
To prove item (b), note that 


LAD = fX) + Vaf, X) © (X* - X), 
and hence 
FX - f,(X) = aV yf, X) © AX. 
Since 
Vxh(X) = —Vigf X)AX, 
we have 
FX») - f,(X) = -a AX o Vef, (X)AX < 0, 


where the strict inequality follows from the positive definiteness of the Hessian matrix 
Vi fX ) (as f, is strictly convex). Thus, f,(X*) < f,,(X). The proof is complete. a 
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12.5.3 Complexity Estimates 


In this part, we analyze the complexity of the proposed path-following algorithm for SDP. 
More specifically, we prove that the iteration-complexity of Algorithm 12.1 is bounded by 


O(vnin [eX e S]). 


Our proof depends essentially on the following two lemmas. 


Lemma 12.7 Let (X,y,S) © Fyicnp X Foigpp» and (X, y., S) be obtained by applying scaling 


to (X,y, S) with H = cuI — XS, and (AX, Ay, AS) be a solution of System (12.18). For any 
a E R, we set 

(X(a), (a), S(a)) Ê (X, y, S) + a(AX, Ay, AS), 

u(a) Ê ZX (0) © S(a), 

V(a) Ê X(a)e S(a) — uI. 
Then 


V(a) =(1—a)(X ° S — uI) + 0? AX ° AS. (12.19) 


Proof: See Exercise 12.2. E 
Lemma 12.8 Let(X,y, S) E€ Fyignp X Fpispp and (X, y. S) be obtained by applying scaling 
to (X, y, S) such that ||XS — „uI || < 0n, for some 0 € [0, 1) and „u > 0. Let also (AX, Ay, AS) 
be a solution of System (12.18), H = oul — XS, ôy Ê u||AXX]|lp, ôs = |IXAS]|p. Then, 
we have 


2 

Wd Al 
by55 < = (65 +65) < ————.. 12.20 
x55 < 5 (ôx + s) $ op ( ) 


Proof: See Exercise 12.3. a 


The following theorem analyzes the behavior of one iteration of Algorithm 12.1. This 
theorem is due to (Touil et al., 2017, Theorem 4.6). The result here in SDP is the counterpart 
of that in Theorem 11.4 for second-order cone programming, and its proof is also the same 
as the proof of its counterpart. 


Theorem 12.4 Let® € (0,1)and ô € [o. yn ) be given such that 
6? +6? 
-oz ([]- 
2(1 — 6) (1 +.) 


Suppose that (X, y, S) E Nepp() and let (AX, Ay, AS) denote the solution of System (12.18) 
with 


(12.21) 


H =opl —XeS and o = 1 — ô/yn. 


Then, we have 
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(a) X+ e S+ = (1 — 8/yn)X eS. 
(b) (X+, y+, S+) = (X,y, S) + (AX, Ay, AS) E€ Npp H). 


Proof: Item (a) follows directly from item (c) of Lemma 12.6 with a = 1 and o = 1 — ô/ Jn. 
We now prove item (b). Define 


wes - XteSt=(1-6/n)u, (12.22) 
and let (X, y, S) E Nspp(w), we then have 
loul -Xes < Io- DaI + Wl — Xe Slip 


= 12.2 
< ((@-1)r + 6) p = (6? + 67) p. 9) 
Since ||X eS — „I|| < Ou and H = onl — X ° S, using Lemma 12.8 it follows that 
oul — X ° S|? 
Axx] -IXAS||- < iaa (12.24) 
2(1 — 0)’ u 


Defining V+ ê V(1) = X* o S* — u*tI and using (12.19) with «æ = 1, (12.24), (12.23), 
(12.21) and (12.22), we get 


oul—XeS|\2 (624+ 62 
Vp = AXAS|p < JAXX" I] 1XAS||p < UM ug Oe 


21 -6)24 ` 20-6) 
<d{1- 2 H. 
yn 
Consequently, 
|X* o S+ — pl ||, < Op". (12.25) 


By using the right-hand side inequality in (12.20), and using (12.23) and (12.21), we have 


lloul — X° Slip p y 62 + 02 z 


=i 
ja Ne aa ay 


where the strict inequality follows from 6 < ; and0Q<1- a <1. 
One can easily see that || AXX~'||,, < 1 implies that 7 + AXX~' > 0, and therefore 
X* = AX +X = (I+ AXX“) X > 0. 


Note that, from (12.25), we have Amin (X* o S+) = Amin (X*S*) > (1 -0)u* > 0, and 
therefore XtSt > 0. Since X+ > 0 and X+ and S+ commute, we conclude that S+ > 0. 
Using the first equation of System (12.18), we get 


AX* = A(X + AX) = AX +AAX =b, and hence Xt € Feonp. 


By using the second equation of System (12.18), we get 
Atyt + St = A* (y + Ay) + (S + AS) = A*y + S+ A* Ay + AS =C, 


and hence (yt, S+) € Ffişpp Thus, in view of (12.25), we deduce that (Xt,y*,St) € 
N gpp(u). Item (b) is therefore established. The proof is now complete. E 
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Now we present Corollary 12.2 in SDP to be the counterpart of Corollary 11.2 for second- 
order cone programming. The proof of Corollary 12.2 is also the same as that of its counter- 
part. 


Corollary 12.2 Let and ô as given in Theorem 12.4 and (X°, y?, S) E€ Nopp(u). Then 
Algorithm 12.1 generates a sequence of points {(X, y®, S®)} C Ngpp(m) such that 


k 
XY os = (: = +) XO eS, Yk>0. 
n 


Moreover, given a tolerance e€ > 0, Algorithm 12.1 computes an iterate ((X,y®, s®)} 
satisfying X® e S® < ein at most 


o(vim (5) 


iterations. 


Proof: Looking recursively at item (a) of Theorem 12.4, for each k we have that 


k 
X os = (: - +) XO eS <e. 
n 


By taking natural algorithm of both sides, we get kln(1 — 6/y/n) < In(e/(X® e S)), 
which holds only if 


(5) "ewes 


or equivalently, 


n (0) (0) 
an a (2282), 


2—1 
ô € 
The proof is complete. E 


In this section, we have presented and analyzed a path-following algorithm for SDP by 
extending the path-following algorithm that was presented in Section 11.5 for second- 
order cone programming. Exercises 12.5-12.8 aim to derive a homogeneous self-dual 
algorithm for SDP by extending the homogeneous self-dual algorithms that were pre- 
sented in Section 10.7 for linear programming, and in Section 11.6 for second-order cone 
programming. In fact, there are several interior-point methods that can be extended from 
linear programming and second-order cone programming to SDP. In this context, we end 
this chapter by presenting Table 12.1, which shows gradient and Hessian derivatives for the 
logarithmic barriers applied on the most three well-known examples of conic constraints: 
the polyhedral, second-order cone and semidefinite constraints. 
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Table 12.1 Comparison of gradient and Hessian derivatives of the logarithmic barrier for polyhedral, second-order cone, and semidefinite constraint 


Polyhedral cone? Second-order cone”) 


Semidefinite cone® 


s, 
Space R” e-{e=(*):senxen} 
5 


S” = {S Ee R®” : S= S1} 


Cone R} = {s€ R” : s > 0} E} = {s € E” : sọ > IIšII} Si = {S ES”: S20} 
Feasibility constraint s(x) = Ax -b € R} s(x) = Ax — b € E} S(x) = $x -BeES} 
i 
Logarithmic barrier / (x) —1" In s(x) — In det (s(x)) — In det (S(x)) 
Gradient V,.7(x) —A's7! -A's —A'vec(S-}) 
Hessian V2, / œ) A'S?A AQ nA AS 8 S)A 
a) [Notations for specifying derivatives in R} ]: We use 1 to denote the vector of all ones of appropriate dimension. For any strictly positive vector s € R”, we def 
Ins ê dns, ..., ln s)" ands! ê Go rs a We use S ê diag (s) to denote the n x n diagonal matrix whose diagonal entries are s,,... ,5,,- 
b) [Notations for specifying derivatives in E} ]: All these notations have been introduced in Table 11.1 including the arrow-shaped matrix and the quadratic 
representation. 
c) [Notations for specifying derivatives in S} ]: For two matrices S, T € S”, the Kronecker product S @ T is the n? x n? block matrix whose i, j block is 


sjT, i,j = 1,...,n. The vectorization of a matrix S, denoted vec(S), is the vector obtained by stacking the columns of S on top of one another. We use A to den 


the matrix whose ith column is vec(A;). 


Exercises 


Exercises 


12.1 


12.2 


12.3 


12.4 


12.5 


12.6 


12.7 


12.8 


Prove item (c) in Lemma 12.6. 
Prove Lemma 12.7. 
Prove Lemma 12.8. 


Implement Algorithm 12.1 and test it on the pair (PISDP) and (DISDP) with 
C = Diag(5;8;8;5) E€ SÍ, b ê(1;1;1;2 E€ Rf, A, êI esf, 


and A,’s, i = 1, 2, 3, are so that 


1, if j=k=i, orj=k=i+1, 
-1, if j=i,.k =j+1, orf =i+1,k =i, 
0, otherwise. 


(4) x 2 


Take X = HI e St, y® = (1.5; 1.5; 1.5; 1.5) € R4, and 


2 150 0 
1.5 3.5 1.5 0 
0 1.5 3.5 1.5 
0 0 15 2 


So = 


as your initial strictly feasible points. You may also take e = 10-°,o = 0.1, and 
p = 0.99. 


In this and the following exercises, we practice extending interior-point methods 
from linear programming and second-order cone programming to SDP. In particular, 
this and the following exercises aim to derive a homogeneous self-dual algorithm 
for SDP by extending the homogeneous self-dual algorithms that were presented 
in Section 10.7 for linear programming, and in Section 11.6 for second-order cone 
programming. In this exercise, we ask the reader to write the homogeneous model 
for the pair (PISDP) and (DISDP). 


Write the search direction system corresponding to the homogeneous model 
obtained in Exercise 12.5. 


State the generic homogeneous algorithm for solving the pair (PISDP) and (DISDP) 
based on the search direction system obtained in Exercise 12.6. 


Let €p > 0 be the residual error at a starting point, and e > 0 be a given tolerance. 
Estimate (without proof) the computational complexity (worst behavior) of the algo- 
rithm obtained in Exercise 12.7. This computational complexity must be in terms of 
the rank of the underlying positive semidefinite cone (n). 
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12 Semidefinite Programming and Combinatorial Optimization 
Notes and Sources 


SDP is a branch of mathematical optimization that emerged in the late 20th century. It was 
initially developed by mathematicians and computer scientists as an extension of linear 
and convex programming. The concept of SDP can be traced back to the works of Leonid 
Khachiyan, who introduced a polynomial-time algorithm for solving linear matrix inequal- 
ities in 1979; see Khachiyan (1979). The pivotal moment for the formalization of SDP was 
the development of the interior-point method for SDP problems by Yinyu Ye in the 1990s 
(refer to Ye (1997)). Since then, SDP has found numerous applications in optimization, con- 
trol theory, combinatorial optimization, and other fields due to its ability to handle a wide 
range of complex optimization problems. 

In this chapter, readers delved into semidefinite optimization problems, studied some of 
their combinatorial applications, and explored the concept of SDP duality, shedding light 
on the intriguing interplay between primal and dual formulations. Furthermore, readers 
studied efficient primal—dual methods that had been developed for solving SDP problems, 
providing readers with valuable tools to address complex optimization challenges across 
different disciplines. 

As we conclude this chapter, it is worth noting that the cited references and others, 
such as Boyd et al. (2004); Ben-Tal and Nemirovski (2001); Renegar (2001); Davidsson 
(2013); Blekherman et al. (2013); Wolkowicz et al. (2000), and Aliprantis and Tourky 
(2007), also serve as valuable sources of information pertaining to the subject matter 
covered in this chapter. The code that created Figure 12.2 is due to StackExchange (2018). 
Exercises 12.5-12.7 are due to Potra and Sheng (1998). 
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Appendix A 


Solutions to Chapter Exercises 


1.1 (a) (i). (e) 6). (i) (iv). (m) (i). 
(b) (ii). (f) Gi). Q) (iii). (n) @). 
(c) @. (e) (ii). (k) ii). (0) Gii). 
(d) ©. (h) (iv). (1) (iv). (p) (ii). 
1.2 (a) A true proposition. (f) A true proposition. (k) Not a proposition. 
(b) Nota proposition. (g) Not a proposition. (1) A true proposition. 
(c) Nota proposition. (h) A true proposition. (m) Not a proposition. 
(d) A true proposition. (i) Not a proposition. (n) Not a proposition. 
(e) Nota proposition. (j) Not a proposition. 
1.3 (a) Today is not Thursday. (g) The summer in Santiago is hot and 
(b) There is pollution in New Jersey. humid. 
(©) 2+143. (h) The sun is shining in Santiago’s sky 
and I am not going to the nearest 


(d) Sara’s first answer to item (J) in 
beach. 


(i) The sun is shining in Santiago’s sky 
and I am not going to the nearest 
(f) The summer in Santiago is not hot or beach or not doing a little physical 


not bearable. exercise. 


Exercise 1.1 was correct. 


(e) The summer in Santiago is hot. 


1.4 (a) Idid not buy a lottery ticket this week. 
(b) I bought a lottery ticket this week or I won the million dollar jackpot on Friday. 
(c) Ibought a lottery ticket this week and I won the million dollar jackpot on Friday. 
(d) Idid not buy a lottery ticket this week and I did not win the million dollar jackpot 
on Friday. 


1.5 (a) PAQ. (b) PAQ. (c) aP AnQ. (d) PVQ. 


Combinatorial and Algorithmic Mathematics: From Foundation to Optimization, First Edition. Baha Alzalg. 
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd. 
Companion website: www.wiley.com/go/alzalg 
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1.6 


1.7 


1.8 


1.9 


1.10 


1.11 


e Ifyou have a passing score on the final exam, then you will receive a passing grade 
for the course. 

e You will receive a passing grade for the course if you get a passing score on the 
final exam. 

e Receiving a passing grade for the course is necessary for getting a passing score on 
the final exam. 


Getting a passing score on the final exam is sufficient to receive a passing score for 
the course. 

e You will get a passing score on the final exam only if you receive a passing grade 
for the course. 


(a) Weare on the line TF, so it is false. (d) We are on the line FF, so it is true. 
(b) We cannot be on the line TF, so it (e) Weare on the line TT or the line FF 


is true. only, so it is true. 
(c) We cannot be on the line TF, so it (f) We are on the line TF, so it is false. 
is true. (g) We are on the line TT, so it is true. 


Restating the theorem gives us “If x and y are odd integers, then their sum x + y is 
even.” From here, all odd integers can be represented as 2k + 1 for some integer k. 
That is, 


x=2n+1 and y= 2m + 1 for some integers n and m. 
It follows that 
x+y=(2n+1)+(2m4+1)=2(m4+n+1). 


Given that m and n are integers (m + n + 1), is also an integer. Thus, x is an integer 
multiplied by 2, hence it must be an even integer. 


The implication statement is “If the Sun is shrunk to the size of your head, then the 
Earth will be the size of the pupil of your eye.” So the contrapositive is “If the Earth 
will not be the size of the pupil of your eye, then the Sun is not shrunk to the size 
of your head,” the converse is “If the Earth will be the size of the pupil of your eye, 
then the Sun is shrunk to the size of your head,” and the is inverse “If the Sun is not 
shrunk to the size of your head, then the Earth will not be the size of the pupil of 
your eye.” 


(a) Any statement “If P then Q” for which P and Q are both true or both false. Take, 
for instance, the statement “If 1 +1 = 2 then 2- 1 = 1.” 
(b) Impossible. 


e The contrapositive of this theorem is “If x is odd, then x? is odd.” An odd integer 
can be represented as 2n + 1, where n is some integer. 


Now x = 2n + 1, which implies that x? = (2n + 1)? = 2(2n” + 2n) +1. 


Given that n is an integer (2n? + 2n), must also be an integer. This means that x? 
must also be an odd integer, proving that the theorem is true. 


Appendix A Solutions to Chapter Exercises 


e The theorem says that “x* is odd + x is odd.” 


(=) The part that “If x is odd, then x? is odd” was already proved in item (a). 
(<) To prove the part that “If x? is odd, then x is odd,” note that the contrapositive 
of this statement is “If x is even, then x? is even.” Given that n is an integer, an 


even number can be represented as 2n where n is some integer. 


Now x = 2n, which implies that x? = (2n)? = 4n? = 2(2n?). 


Given that n is an integer, 2n? must also be an integer. This means that x? must 


also be an even integer, proving that the theorem is true. 


1.12 (a) [P< Q] 


(b) ~P @ Q] 


AP > QAQ > P)] 
(P > Q) V ~Q > P) 
ACP V Q) V aQ V P) 


(aP A =Q) V (nQ AnP) 


(P AQ) V (QA =P). 


[P V Q) ATP AQ) 
(P V Q) V AAP AQ) 


= (PV Q) V PAQ) 


(GP A7Q)V (PA Q). 


1.13 We construct the following truth table. 


(The hint) 
(DeMorgan’s law) 
(Implication law) 
(DeMorgan’s law) 
(Double negation law) 


(The hint) 
(DeMorgan’s law) 
(Double negation law) 


(Double negation law) 


Hadad 
Homan 


P>Q Q>P PH-QAQ->P) 


Homan 
Hams 
Bom s 


Because the truth values for P ~ Qand (P > Q) A^ (Q > P)are identical, this implies 


that they are logically equivalent. 


1.14 (a) Below is “P A =P” truth table. 


P =P 


T F 


P 


F 


AnP 
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(b) Below is “(P v ~Q) > Q” truth table. 


P Q =Q Pvg) (PV =Q) > Q 


eo eS a | 
Harg 
r r 4 ES 
e eo e H 
oH i A 


(c) Below is “(P v Q) > (P A Q)” truth table. 


P Q PVR PAQ (PVQ)>(PAQ) 


yA 4 4 
jag 
es er ee | 
Wy sy 4 
Hmm #A 


(d) Below is “(P > Q) e (“Q > ~P?” truth table. 


P Q aP =aQ P>Q 3Q > ~P (P > Q) (AQ > AP) 


Ama A 
jarag 
e e e: ea 
ar m a Fc 
ar iiiar e s ae 
Harg 
ar See ae ae 


(e) Below is “P @ P” truth table. 


P PO@P 


T F 
F 

(£) Below is “P ® ~Q” truth table. (g) Below is “=P ® ~Q” truth table. 
P Q =Q P@®-Q P Q 7P =Q ~P 


H jaAa 
Aam A 
or eS Oe | 
am HA 
Ama A 
go e Ee: 4A 
aoa e e 
ioe eo e e 
z: UE e ME e BRS 


(h) Below is “(P @ Q) A (P @ 7Q)” truth table. 
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P 


Wma A 


Q 


Aa A 


=Q 


ar e E e 


POQ 


Ho a 


P@®-Q 


Om m AH 


(P E Q) A (P OQ) 


oo g a eee 


(i) Below is “P —> ~Q” truth table. 


P 


H o HA 


Q 


T a mi a 


aQ 


H I a 


(j) Below is “~P + Q” truth table. 


(k) Below is “(P > Q) v (AP > Q)” truth table. 


P 


les iiie E E 


is E e EE 5 E 


don o 


=Q 


ar iieo iar Fw 


P>Q 


Harag 


P Q =P Po 
T T F F 
T F F T 
F T T T 
F F T F 
mP >Q (P > QYVOP> Q) 
T T 
T T 
T T 
F T 


(1) Below is “(P Q) v (AP © Q)” truth table. 


as) 


Ho a 


Q 


s B e ie s 


doin ee 


H a a A 


(Po QV (AP © Q) 


ar ie ae ae 
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(m) Below is “(P A Q) v R” truth table. 


P 


Aum A AAA HA AH 


Q 


Ama AHA wD DAH 


R 


zs Eim e s e wae es e 


PAQ 


H a es a a a a A 


(PAQ)VR 


ec? a ae | 


(n) “(P A Q) A R” truth table. 


(0) Below is “(P v Q) v R” truth table. 


P Q R (PAQ)AR P Q R PVQ (PVQ)VR 
T T T T T T T T T 
T T F F T T F T T 
T F T F T F T T T 
T F F F T F F T T 
F T T F F T T T T 
F T F F F T F T T 
F F T F F F T F T 
F F F F F F F F F 
(p) Below is “(P A Q) V aR” truth table. 

P Q R aR PAQ (PAQ) VaR 

T T T F T T 

T T F T T T 

T F T F F F 

T F F T F F 

F T T F F F 

F T F T F T 

F F T F F F 

F F F T F T 


(q) Below is “P > (~Q v RY” truth table. 


P 


oes eo aAa a Aaa 


Le) 


AAA A DW DPA A 


R 


Tam HA mA TAH 


=Q 


AAA AHH a 


(-Q Vv R) 


qt aA ey A 
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P > (QVR) 


ar ee ee ae ae oe > ar 


(r) Below is “(P > Q) v (=P > Rọ” truth table. 


P 


T a a a a a a a 


Q 


ies Sies E e e = e > e 


R 


pE ae ee RDA FAH 


aP 


e e e r e e e 


P>Q 


Yodo fw HH 


aP>R 


eo ie eS ae ae oe Be a 


(P => QV CP > R) 


le ie ee Oe er ar or | 


(s) Below is “(P 


=> Q) A (AP > RY” truth table. 


P 


Am Waa AH 


Q 


dtd aA mas 


R 


DHA AHA AHA 


=P 


AAA DTD 


P>Q 


YHA A Se eH A 


aP>R 


Da mtHeHA HH AB 


(P > QACP > R) 


se a a a > a | 
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(t) Below is “(P © Q) v (~Q © R)” truth table. 


P Q R Q P#Q eR PEQVHQR) 


co Mie. ie. e. e e e 
co iie- e e e> e | 
co Me e. e e> e e 
Hat te e e s> 
=e e: e: e aA 
To M e e: e e a o 
HAs PTA e 


1.15 Let F be “Lillian is forceful,” G be “Lillian will be a good executive,” E be “Lillian is 


efficient,” and C be “Lillian is creative.” Then the problem statement can be formu- 
lated in the following propositional logical model. 


FVC, 
FOG, 
(E A^ C), 

aE > (FV 6). 


Problem model: 


The desired conclusion is to conclude that the following propositional formula 
[FV C)A(F > GAGEA ACE > (Fv G))] — G, (A.1) 


is a tautology. Otherwise, we cannot conclude that Lillian will be a good executive. 
Using the DeMorgan’s law and double negation law, the conditional statement (the 
fourth proposition) in (A.1) can be written as 


[FV C)A(F > G)AW(EAC)A(EV FV G)] — G. (A.2) 


Now, our objective is to establish whether the conditional statement presented in 
Equation (A.1) possesses the property of being a tautology or not. The most straight- 
forward approach to ascertain this characteristic is by constructing a truth table. 
Nevertheless, it is essential to acknowledge that many rows in this truth table can 
be omitted from our analysis due to the fact that the only circumstance under which 
the formula can yield a “false” result is when the variable G is assigned a “false” 
value, and concurrently, all the components on the left-hand side of the implication 
are assigned “true” values. Therefore, if we can successfully demonstrate that for 
each and every conceivable combination of values where G is “false,” the entirety 
of the expressions on the left side of the implication is also “false,” we can conclude 
that the implication in the equation will invariably hold true. 
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a) (2) (3) (4) 
F E C G FVC FSG -A7EAC) (EVFVG) DAGDAG 
F F E F F T T F F 
F F T F T T T F F 
F T FE F F T T T F 
F T T E T T F T F 
T E E FEF T F T T F 
T F T F T F T T F 
T T E F T F T T F 
T T T F T F F T F 


As you can see from the table, for every combination of values where G is false, every- 
thing to the left of the implication is also false. This means the original implication 
in (A.1) can never be false, so G must be true. Thus, Lillian will be a good executive. 


1.16 Implication law, associative law, commutative law, distributive law, contradiction 
law, domination law, and idempotent law. 


1.17 All the given implications are tautologies as shown below. 
(a) [7PA(PVQ)] > Q = [PAP VOP] > Q 
= [FV CPAQ)] > Q 
= (PAQ) > Q 
= ~-PAQ)VQ 
= (PV -=Q)VQ 
= PV(-“QV Q) 
= PVT=T. 


(b) IP > Q) A (Q > R)] > (P > R) = ACPVQOACQVR)] V OP V R) 
= [OPV Q)V AQ R] V (AP V R) 
= [P AQ) V (QAR) V (AP V R) 
= (PA7Q)V(QA7R)V>PVR 
= [((PA7Q) VP] V (QAR) VR] 
= [(PV-7P)A (“QV 7P)] Vv (QV R) 

x A (ARV R)] 

= [TACQV->AP)] VKQV RAT] 
= QVP) Vv (QV R) 

(7QV Q)V (AP V R) 

TV (nPVR)=T. 
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(c) PAP > QI >Q = [PACPVQ]1>Q 
= (PAP) V(PAQ)] > Q 
= [FV(PAQ)] > Q 
(PAQ)>Q 
PAQ 
(“PV =Q) VR 
“PV QVQ) 
aPVT=T. 


(d (PVQA(PSRAQSRISR = ADPVQACPVR)A(-QVR)IVR 
= [PV Q)V7CP VV R) 
XVAAQVR)J VR 
= (PAQ) V (PAAR) V (QATAR) VR 
= (P A 7Q) V (PAAR) v (QV R) 
xA GRyYR) 
= (GPA 7Q) vV (PAAR) V [(QV R)AT] 
= (APA 7Q)V(PAAR)VQVR 
= [CP A nQ) v Q] V [(PA TR) v R] 
= [PYAAR QO) V LP VR) 
XA (AR V R)] 
=(OPVQ)ATIVIPVRAT] 
= (Pv Q)V (PVR) 
= ("Pv P)v (QVR) 
=TV(QVR)=ET. 


1.18 Starting with item (a), we have 
P&R =E= P> QAQ -> P) 
= PV Q)A(7Q V P) 
= [OPV Q An] V [CP V Q) A PI 
= [CP An) V (QA AQ) V [CP AP) V (Q A P)] 
= [CP Ang) V F] V [F V (QA P)] 
= (=PA7Q) v (Q A P) (This is item (b)) 
AP V Q)V (QAP) 
= THC V Q)V(QAP)] 
= -[(PV Q)A7(QAP)] = -[P @ Q]. (This is item(c)). 


1.19 
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To construct a DNF having the given truth table, note that 


P Q R Statement Equivalent to 


T T F T PAQAZRAR 
F T T PA7AQAR 
F T F T aPAQAAR 


A DNF would be (PA QA7AR)V(PA7AQAR)V (APA QAZWR). 


(a) -(A V B) = 7A ^ ~B (CNP). 
(b) -(4 A B) = ~A v AB = (nA V 7B) A (A V ~A) (CNP). 
(c) AV(BAC) = (Av B) A (A v C) (CNP). 


(PAaPAQ) V (QA 7Q) V (RA TR) V (PATQA TRA AP). 


Note thatPA QA (ARV SV AT) = (PA QATAR) V (PA QAS)V (PA QANT), which 
is in DNF. According to Remark 1.5, the given proposition is satisfiable. 


(a) (iii). (b) (iv). (c) (ii). (d) (iv). (e) (ii). (f) (iii). 


(a) False. The value x = —1 can make this proposition false. 

(b) False. It is true only if we take x = 0, but this is not in the domain. 
(c) True. The value x = 2 can make this proposition true. 

(d) False. The value x = —1 can make this proposition false. 

(e) True. The value x = 1 can make this proposition true. 


Note that N is defined as N £ {1,2,3,...}. 


(a) Take Pœ, y) = “x +y =x +y.” (c) Take P(x,y) = “xy = y.” 
(b) Take P(x,y) = “xy = z” (d) Take P(x,y) = “xy = x.” 
The negation of the given proposition is Vx € R, 3y € R,x + y  Oas itis seen below. 
~a(3x € R, Vy E R,x +y = 0) = Yx E R, ~a(Yy E R,x + y = 0) 
= Wx € R, 3y E R, ~(x + y = 0) 
=VxER AER X+yY FO. 


Denote by D the set of all people. 

(a) ax € D, AL (x). 

(b) Ax € D, (I(x) A 7C(x) A nA). 

(c) Vx € D, (A) v Cœ] AIX) > L). 


The negation of the sentence “Some children do not like mimes” is the sen- 
tence “Every child has some mime that he/she likes,” which can be symbolized 


431 


432 


Appendix A Solutions to Chapter Exercises 


2.1 


2.2 


2.3 


as Vx E D, [Child(x) > (ay € D, [Mime(y) A Like(x,y)])]. As another approach, 
the quantified statement “Some children do not like mimes” is symbolized as 
dx € D, [Child(x) A (Vy € D, [Mime(y) > 7Like(x, y)])], which can be negated as 
Vx € D, [Child(@x) > ~(Yy € D, [Mime(y) > 7Like(x, y)])]. 


We let D, be the set of all houses and D, be the set of all owners. We also define the 
following two predicates: OWN(v, x) = “The house v has owner x,” and ADJ(v, u) = 
“The house v is adjacent to house u.” 

(a) Vx € D,, w € D}, OWN(v, x). 

(b) Ww € D,, Vu € D}, Vx € D,, (ADJ(v, u) A OWN(v, x) —— ~OWN(u, x)). 

(c) Ww € D,, Yx € Dy, (OWN(v, x) —— [73y € D,, (y # x) AOWN(v, y))]). 


Denote by D the set of all people. We also define the following three predicates: 
Fr(x, y) = “The two persons x and y are friends,” Sm(x) = “The person x smokes,” 
and Ca(x) = “The person x has a cancer.” 

(a) Yx € D, Yy € D, Yz € D, (Fr(x,y) A Fry, z) —— Fr(x, Z)). 

(b) Yx € D,(=[3y € D, Fr(x, y)] — Sm). 

(c) Yx € D, (Sm(x) ——> Ca(x)). 

(d) Yx € D, Yy E D, (Frx, y) —— [Sm(x) —- Sm(y))). 


(a) (iii). (© (iv). (e) (iv). (g) (ii). (i) Gi). 
(b) (iii). (d) (ii). (f) (iv). (b) (iii). Q) (iv). 


Suppose the principle is true when n = k, for some k € N. Then there is some natural 
number s such that m < sk. Choosing the same s, we have m < sk < s(k + 1), so the 
principle is true when n = k + 1. The proof is complete. 


By induction, let P(n) be 
2_ nnt+1)2n+1) » 
A ; 


First, we look at the base case for n = 1 to see whether or not P(1) is true. Substituting 
n = 1 into both sides of the equation, we see 


“142? 4+3°+---4+7 (A.3) 


1, 


ve te (11 +12 *141) _ 
i 6 
i=1 


is true. Now, we look at the inductive step for n = k to see whether or not P(k + 1) is 
true. By substituting all n’s with (n + 1) on the right side of (A.3), we get 


(n+1)(n+1)+1QM4+1)+1) _ (nt+1)(n+2)2n +3) 
6 6 
Z (n? + 3n + 2)(2n + 3) (A4) 
6 
2n? + 9n? + 13n +6 
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If the inductive hypothesis P(k), which is 
“7427 432 4..-4 P= ——— 
is true, then P(k + 1), which is 
k(k+1)2k+1 
14243? 4+---+h +(k+1P = SOE e+? 
_ kk+1)2k +1) z 6(k + 1}? 
6 6 
k(k +1)(2k + 1) + 6(k + 1)? 
6 
(k? + k)(2k + 1) + 6(k + 1}? 
6 
2k? + 9k? + 13k + 6 
6 
is also true because this matches in (A.4) that we found by substituting (n + 1) into 
the right-hand side of (A.3). We conclude if P(k) is true, then P(k + 1) is true for any 
k > 1 which proves the inductive conclusion. 


> 


Define the following predicate: 


P(n) = “The cardinality of the powerset of a finite set A is equal to 2” if the cardinality 
of A is n.” 


Base case (for n = 0): The set A with cardinality 0 is the empty set Ø. Its powerset 
(set of all subsets) is Ø. Since P(A) = {Ø}, we have |P(A)| = 1 = 2°. Therefore P(0) 
is true. 


Inductive step: Assume that P(k) is true. To prove that P(k + 1) is also true, let B be 
any set of cardinality k + 1. Enumerating the elements of B as a4, a3, 03, ... , Ak, Ak41- 
That is, B = {a,,d,,d3,...,@,,a,,,} and |B] = k + 1. Define 
A = {Q),Q5,03,...,a,}, 

then |A| = k and |P(A)| = 2* (by the inductive hypothesis). 

Note that B = A U {a,,,} and that every subset of A is also a subset of B. 

Now any subset of B either contains a,,, or it does not. Every subset C (of B) that 
does not contain a,,, is also a subset of A, of which there are 2* (by the inductive 
hypothesis of P(k)). For every subset C not containing a,,,, there is another subset 
of the form C U {a;,,,} containing it. Since there are 2 such possible subsets C, there 
are 2* subsets of B containing a,,,,. We now have that B has 2* subsets not contain- 


ing a,,, and 2k containing it. Therefore, |P(B)| = 2* + 2¥ = 21 and hence P(k +1) 
is true. 


Thus, by induction on n, P(n) is true for all n € N. 


The base case for n = 1 is seen to be true as T(1) = 1 = 3! — 2 = 1, where the first 
equality follows from the piecewise function. Now, assume that the inductive 
hypothesis holds for n = k, that is, T(k) = 3* — 2. Then 


T(k +1) = 3T(k) + 4 = 3(3* — 2) + 4 = 3(3")-2 = 3"! — 2, (A.5) 
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2.6 


2.7 


2.8 


2.9 


2.10 


3.1 


3.2 


Therefore, if T(k) = 3" — 2, then T(k + 1) = 3**! — 2 for an integer k > 0. This proves 
the inductive conclusion. Thus, by induction on n, the proof is complete. 


(a) False. To see this, let a be any real number, and consider A = {a}, B= {{a}}, 
and C = {{{a}}}. Then A € Band B E C, but A ¢ C because C only contains one 
element which is not A. 

Note that the statement becomes true if the symbol membership symbol “ce” is 
replaced with the inclusion symbol “C.” That is, the following statement is true. 


AcRABEO— Aco. 


(b) False. Take, for example, A = {1,2}, B = {1,3}, and C = {2, 3,4}. 


(a) Since |S| = 5, the number of subsets of S is 2!5! = 25 = 32. 
(b) Note that T = {22, 24, 26, ..., 38,40} and that |T| = 10. So, the number of subsets 
of T is 2'7! = 2! = 1024. 


(AUC’)NB' = {u,w,r, t}. 


We have to show that the relation & is reflexive, symmetric, and transitive. For 
reflexivity, each person x has the same birthday of him/herself. For symmetry, if 
x and y have the same birthday, then y and x have the same birthday. For transitiv- 
ity, if x and y have the same birthday and y and z have the same birthday, then x 
and z have the same birthday. Thus, the relation & is an equivalence relation. Since 
there are 365 ways two people can have the same birthday, & has 365 equivalence 
classes. 


The exponential function f : R > [0, œ) defined by f(x) = e” is an injection. To 
see this, let x,y E€ R such that e = e’, then x = lIn œ = Ine” = y. Now, we show that 
the function f is a surjection. To see this, note that any y € [0, oo), we can choose 
x = Inyand have f(x) = e” = e" = y. Therefore, f is both an injective and surjective 
function. Thus, f is a bijection. 

The exponential function g : R > R defined by g(x) = e* is not a bijection because 
it is not a surjection. For instance, because e* > 0 for any x € R, there is no x € R 
such that g(x) = —1. 


(a) (iii). (c) (ii). (e) ©. (g) 6). 
(b) (iv). (d) @. (f) (i). (h) ©. 


Let S be the set of positive integers for which the statement is true. Since 
k 
lim an =0, 1 € S. Assume that k € S, that is, lim, E Using Hospital’s 


noo n ` 


rule, it follows that 


k+1 k 
lim BP - jim ELU _ (k+1)lim 


n>% n n>o h n>% 


=0. 


(Inn) 
n 


Thus, k + 1 € S. This proves the result by induction. 
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3.3 Using Table 3.1 (see also the result in Exercise 2.3), we have 


60 60 29 
Sie = Sue Se 
k=30 k= { 

60-61-62 29-30-31 
6 6 
10 -61-62—29.5.31 


37,820 — 4495 = 33, 325. 


3.4 Note that 
1 _ (2k+1)/2-(2k-1)/2_ 1/2 1/2 
Qk+DQCk-1) Qk+DQk-1)  2k-1 2k+1' 


The sequence of partial sums is 


n 


1 
Se= 2 (2k + 1)(2k — 1) 


<a f 1/2 1/2 
ye a) 


/2 1/2 1/2 1/2 1/2 1/2 
“(Cre a (P-P) (4-54) 
1 


1 
2 2n L 1 2: 
Thus, the series converges to 1/2. 


3.5 (i) Note that lim,_,.,3-* = lim,_,,,1/3* = 0. It follows that 
s J ii 1 
lim ——— = — = 5. 
k>% 72 + 37k 2+0 2 
Using the divergence test, the series Xg; = = diverges. 


(ii) Using Theorem 3.3 (iv), we have lim,_,,,(0. 3)* = 0. It follows that 
1 1 1 


lim ———_ = —_=-. 
k>œ2 + (0.3% 2+0 2 


Using the divergence test, the series $% diverges. 


1 TT 3)k 
3.6 (i) Since 2/3 < 1, using the geometric series we have 


AOE 


k=l 1 


The series converges to 2. 
Gi) Note that 


ms) =O 


k=1 k=1 


Since 5/4 > 1, using the geometric series, we conclude that the series diverges. 
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(iii) Using the geometric series, we have 


2 2 


Yt =, if <1 
= 


Replacing x with —x, we get 


œ œ 2 


kk ko (x x 
pa 9 = 2h i 1- (=X) 1+x 


k=2 k=2 


When x = 2/5, we have 
` - z 2 
Sut)" = Emr yh 


The series converges to 5/7. 
(iv) Using the geometric series, we have 


co 
Yt =. iff <1. 
1-x 


Differentiating both sides with respect to x, we get 


= d 
Lets 4 (4) = ao" if |x| <1. 


When x = 2/3, we have 


2 k 
De) = Ll osl y, 


The series converges to 9. 


3.7 IfM e R™"” isa symmetric positive definite matrix, and x € R” is a nonzero vector, 
then 


x'M7!x = x! (MMM) x = (M7!x)'MM~!x = y'My > 0, 
where y = M~'x. Thus, M™! is also positive definite. 
3.8 Using Proposition 3.1, we have 


Ht 


Il 
~~ 
tad 
M 
A 

N 
Ea 
| 
N 
Ra 
Il 
=] 
y 


Il ll 
m- m 
p, 

SOR KO X 
| 
M 
aA 
N 
_ 
| 
N 
Ln! 
E 
i 
© 
K~~’ 

* 


3.9 


3.10 


3.11 


4.1 


4.2 


4.3 
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Let x,y E€ R” and å € [0,1]. We have 
fx +A- Ay) = max fi(ax + (1 — dy) 
< max (Afi) +a- afi) 
SA ere ae Ga re 


Af (x) + (1 - AF), 


where the first inequality follows from the convexity of f, i = 1, 2, ... , m. 


Assume that S4, S2, ... , Sm are convex sets. Let x,y € n S; and å € [0,1]. By the 
convexity of S,,i=1,2,...,m, we have Ax + (1 — A)y E S, i = 1,2, ...,m. It follows 
that Ax + (1 — Aly € Nt S; Therefore, N? S; is convex. If an infinite number of con- 
vex sets intersect, the result is also correct and a proof similar to the above one can 
be used. In addition, if the number of convex sets is infinite but countable, a proof 


by induction can be also thought of. 


Under the assumption in Farkas’ lemma (Version II). It is clear that if (2’) holds, 
then so does (2). Now, assume that (2) is true, that is, Ax = 0, c'x < 0 and x > 0, and 
let x = —x/(c'x). Then, the nonnegativity of x follows from the nonnegativity of x 
and the negativity of c'x. Moreover, 


To Tf ale ee =_ygf(zly\y__ S 
cx=c (=) = ae x=-l, and Ax=A(>x) = age 
This proves that if (2) holds, then so does (2’). 
(a) (iv). (d) (iv). (g) ©. 
(b) (ii). (e) (ii). (h) (iii). 
(c) (i). (f) (iii). (i) (Gv). 


A spanning tree so that no vertex has a 
degree of 4 is shown to the right. (Other 
answers might be possible.) 


Yes. To see this, labeling the vertices of G, and that of G, creates a matching between 
them in a way that preserves adjacency as shown below. 


| a ” 
S 
i GH?) 
G2 


Gi 
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4.4 (a) The answer is no, because we have an odd number of odd degree vertices. 


4.5 


4.6 


4.7 


4.8 


4.9 


4.10 


(b) No. The justification for this answer is that there are two four-degree vertices. 
The minimum number of vertices (outside of themselves) that those two 
vertices must be connected to is 6. Therefore, all eight vertices are connected. 
However, the degree sequence has two vertices of degree 0, contradicting 
this. 

(c) Yes. To justify this answer, a graph that 
represents the given degree sequence 
is shown to the right. e 


Assume that we have an acyclic graph denoted as G, and the addition of the edge 
(x,y) to G results in the creation of a cycle for every pair of vertices x and y in the 
vertex set V where (x,y) ¢ E. In order to establish that the graph G is a tree, we 
need to demonstrate its connectedness. Let us take two arbitrary vertices, say u and 
v, within G. If u and v are not already adjacent, the introduction of the edge (u, v) 
leads to the formation of a cycle. However, it is important to note that all edges in this 
cycle, except for (u, v), are part of the original graph G. Consequently, there exists a 
path between u and v within G. Given that our choice of vertices u and v was entirely 
arbitrary, we can conclude that G is indeed a connected graph. 


Only one graph is possible, which is the complete graph K;. 


The graph K, has no a Hamiltonian cycle because the sizes of its partition subsets 
are not identical. The graph K,, has no a Hamiltonian path because the sizes of its 
partition subsets differ by more than one. 


(a) The tight upper bound is k + 1. We may need to add a single color, that is, change 
the color of one endpoint to a new color. 

(b) While k is still an upper bound, the tight upper bound would be k — 1. Let us 
consider worst case which is the complete graph K„. The graph K, is not (n — 1)- 
colorable (Theorem 4.12). Now, removing any edge from K,, reduces the number 
of colors required to properly color it ton — 1. 


The statement is false. From Theorem 4.12, the complete graph K,,, is not (d + 1)- 
colorable. Let v,, v2, ..., Ud, Ug41> Ud42 be the vertices of K,,,. Let also G be the graph 
formed by adding a new vertex u of degree d and adjacent to each of v,, Vz, ..., Ug- 
Then G has a vertex of degree d but it is not (d + 1)-colorable. 


The strongly connected components of the given digraph are surrounded by blue 
dashed polygons in Figure A.1. 
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Figure A.1 The strongly connected components of the 
digraph of Exercise 4.10. 


5.1 (a) (iv). (b) (ii). (c) (iii). (a) (ii). (e) (iv). 


5.2 (a) The recurrence relation T(n) = 2T(n/2) + n is of the merge sort algorithm. In 
Example 5.5, we used the iteration method to solve this recurrence. Now, we use 
the guess-and-confirm method to obtain the same answer. In this recurrence, 
each n should be a power of 2 (i.e., n = 1, 2, 4,8, ...); otherwise, we have either 
noninteger inputs or inputs with undefined references. It is given that T(1) = 1. 
Note that 


T(2) = 2T(1) +2 = 4 = 2log2 +2, 

T(4) = 2T(2) + 4 = 12 = 4l0g4 +4, 

T(8) = 2T(4) + 8 = 32 = 8log 8 + 8, 
T(16) = 2T(8) + 16 = 80 = 16 log 16 + 16, 


T(k) = klogk +k. 


It seems that T(n)=nlogn+n for every n= 2' and i=0,1,2,3,.... We 
prove this by induction on n. The base case is trivial: T(1) = 1 = 1log1+1. 
Assume that the statement is true for all m < k, that is, T(m) = m log m + m 
for any m < k. Now, we prove that T(k) = klogk +k. Since k is a power of 
2, we have k = 2! (and hence i=logk) for some i€ N. Since 2! < k, we 
particularly have T(2'1) = 2'log2'1 +25! = 2) 1@-1)4+2'1 = i(2!) by 
the inductive step. It follows that T(k) = 2T(k/2) + k = 2T(2'1) + k = 2i(2°-}) + 
k = i(2') +k =klogk +k. This confirms that T(n) = nlogn +n. 

(b) The recurrence relation T(n) = 3T(n — 1) + 4 is the one given in (5.1). We use 
the guess-and-confirm method to solve this recurrence. It is given that T(1) = 1. 
Note that 


T2) = 3T0)+4=7=37-2, 
T(3) = 3T(2) +4 = 25 = 3° — 2, 
T(4) = 3T) +4 =79 = 31-2, 
T(k) = 34-2. 


It seems that T(n) = 3” — 2 for n = 1,2,3,.... We prove this by induction on n. 
The base case is trivial: T(1) = 1 = 3! — 2. Assume that the statement is true 
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for n = k, that is, T(k) = 3* — 2 for some k € N. Now, we prove that T(k + 1) = 
3%D _ 2, Note that 


T(k + 1) = 3T(k) +4 = 3 (3-2) +4 = 3 (3f) - 6 +4 = 31-2, 


Thus, T(n) = 3” — 2 for any n = 1,2,3, .... This confirms the solution given 
in (5.2). 


5.3 Using repeated substitutions, we have 


TQ) = 1 = (0 — 3)2? + 4, 


T(2) = 0 = (1 — 3)2! + 4, 
T(3) = 0 = (2 — 3)2? + 4, 
T(3) = 5T(2) — 8T(1) + 4T (0) = 4 = (3 — 3)2? +4, 


T(n) = 5T(n — 1) — 8T(n — 2) + 4T(n — 3) = (n — 3)2" + 4, forall n > 1. 
We prove the nth-term guess by mathematical induction. The base case is trivial: 
T(0) = 1 = (0 - 3)2? +4, T(1)=0 = (1 —3)2! +4, and T(2)=0 = (2 -— 3)2? + 4. 
Assume that the statement is true for all m < k, that is, T(m) = (m — 3)2” + 4 for 
any m < k. Now, we prove that T(k) = (k — 3)2* + 4. This can be seen from the 
following: 
T(k) = 5T(k — 1) — 8T(k — 2) + 4T(k — 3) 
= 5((k — 4)2® + 4) — 8((k — 5)2® + 4) + 4((k — 6)2® + 4) 
= (5k)2"-) — (20)2*-) + 20 — (8k)2* + (40)2% — 32 
+(4k)2"-) — (24)2*- + 16 
= (2.5)(k)2* — (10)2* + 20 — (2k)2* + (10)2* — 32 
+(0.5)(k)2* — (3)2k + 16 = (k — 3)2* + 4. 


5.4 (a) Note that 
T(n) = 5T(n — 1) = 5(5T(n — 2)) = 5(5(5T(n — 3))) = ++ = Tn —k). 


Let k = n, then T(n) = 5"T(0) = 3(5"). 
(b) Note that 
T(n) = 2T(5) +nlogn 
= 2(2T(4) + 5 log(>)) +nlogn 
= 2°T(5) + n(logn — log 2) + nlogn 
= 2°T(5) + 2nlogn—n 
= 2°T(3) +3nlogn—n-—2n 


= 2*T(2) + knlogn — nýi 
= 2°T(=) + knlogn— ska Dk. 
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Let n = 2*, then 


T(n) = nT(1) + nlog’n — Sogn — 1)logn = cn + Znlog’n + snlog n. 


5.5 Let g(x) be the generating function for the sequence {a,,} = {T(n)}. Then 


g(x) = Tix. 


k=0 


Using the recurrence relation, we have 


g(x) = TO + Tx + $ Thx 


a k=2 
= 1+3x+ Ý -Tk - 1) + 6T(k — 2))x* 
k=2 
= 14+3x-x Ý, Tk- Dx"! + 6x? $) T- 2x? 
k=2 S k=2 E 
= 1+3x+xT(0)-x > T(k — Ix" + 6x? £ T(k = 2x? 
5 k=1 re k=2 
=14+4x-x £ T(k)x* + 6x? £ T(k)x* = 1 + 4x — xg(x) + 6X g). 
k=0 k=0 


It follows that 
g(x) = 1 + 4x — xg(x) + 6x7g(x). 


Solving for g(x), we get 


2. 
1 +x-— 6x? 
z 1+4x 
~ (1 = 2x)(1 + 3x) 
S(1-2x)+ 20 +3x) 6 4 14 
~  (1—2x)1 43x) 51-2x 51+3x 


Using the geometric series (5.7), we get 


co co 


_ 6 kx 1% kk 6_1 k| k 
g(x) sD * =a 3)kx £ 5 2 = (=3¥ |x. 
k=0 k=0 k=0 
Ca 
T(k) 


Therefore, T(n) = (1/5) ((6) 2” + (-1)"*13") for any n > 0. 


5.6 Let g(x) be the generating function for the sequence {a,,} = {T(n)}. Then 


g(x) = E Tix. 


k=0 
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Using the recurrence relation, we have 


TO) + Tx + X Tox“ 


g(x) = 
k=2 
= T(0)+ T(x + Tk — 1) + T(k—2))x* 
k=2 
= TO) + Tx + È Tk - Dx" + Ý Tk - 2x" 
k=2 k=2 
= T(0)+ T(x + > T(k — 1) + XY Tk -2 
k=2 k=2 
= T(0) + T(x — xT(0) + oy T(k — 1)x + XST = 2x? 
k=1 k=2 
=x+ x») T(k)x* + x? Dy T(k)x* = x + xg) + xX g). 
k=0 k=0 
It follows that 


g(x) = x + xg(x) +X g). 


Solving for g(x), we get g(x) = x/(x? +x — 1). 
It can be shown that 


~x-x=- == f 7 ; 
1-x-x° =—-(x+a)(x+f), andhence g(x) care =) 


where a = (1+ 4/5) /2 and p = (1 — V5)/2. 
Note that a = —1/f. It follows that 


b I ee Fart, 
p+x 1+x/f l-ax # 
where the last equality was obtained by the geometric series (5.7). Similarly, we also 
have 


[ee] 
Q 
yee 
& +x = 


Thus, we have 


1 O kk Oek S11 pr k 
g(x) = — ax — » px |= — (a — p*)x |. 
(et že) -Že 


Therefore, T(n) = p™) for any n > 0. 


1 n 
a 
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5.7 The recursion tree for the recurrence T(n) = 2T(n — 1) + 1 is shown below. 


1 => 2° 
1 1 572! 


ar MN 
Pett flo oty 


TQ) = TO) TQ) = TO) TQ) © TO) TO) = TA) 


1 1 1 >? 


From the tree, the total cost at level i is (3/2)'n. The height of the tree, h, is obtained 
when n — h = 1. So, we have h = n — 1. It follows that 


n-1 
; 1-2" 

T 2 = —— =2"-1=0(2"). 

(n) 2 T (2") 


5.8 (a) The recursion tree for the recurrence T(n) = T(n/2) + n? is shown below. 


From the recursion tree shown to the right, 
the total cost at level i is n/2”. The height of 
the tree, h, is obtained when n/ 2" = 1.S0,we 
have h = logn. It follows that 


2 ny2 
(5) > (3) 
logn 2 2 


2G)” 


T(n) 


A 
ked 
N 
Me 
fr 
BIR 
= 


Thus, using the asymptotic notation intro- 
duced in Chapter 7, we have T(n) = O(n’). TC) 
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(b) The recursion tree for the recurrence T(n) = T(n — 1) + T(n/2)+n is shown 


below. 
n an 
F , ; 


TA) = TU) TO) = TA) TA) = TA) TO) = TA) 


Since we are finding an upper bound, the height of the tree is decided by the 
longest path, which is the leftmost path. So, the height of the tree, h, is obtained 
when n — h = 1. Hence, h = n — 1. 

Also, because we are finding an upper bound, we can drop the subtracted con- 
stant at each level. Then the total cost at level i can be taken to be (3/2)/n. It 
follows that 


where we used the finite geometric series formula (3.3) to obtain the first 
equality. Thus, using the asymptotic notation introduced in Chapter 7, we have 
T(n) = O(n(3/2)"). 


5.9 (a) The recursion tree for the recurrence T(n) = 4T (2 + 2) + nis shown below. 


n an 
+2 ee) 5+2 3+2 = 2n + 2(4) 
243 243 243 0 E43 243 © 243 3+3 > 5+3 > 2?n4 3(42) 


TO) TOTA) TOTA) «++ TOTA) +++ TATA) + TOTA) «+ TOTA) TOTA) + TA) 
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Although growing, the growth rate of the constant in front of 4' is much smaller, 
so we can ignore it. So, we can take 2'n + 4! to be the total cost at level i. Also, 
the height of the tree, h, can be obtained when n/ 2 = 1. So, we have h = logn. 
It follows that 

logn logn logn 


Tn) > ¥ (2in+4') =n} 2+ Yai 
i=0 i=0 i=0 


Using the finite geometric series formula, we have 


1- 20+logn) 1- qGtlogn) An2 =Ï 10 2 1 
> paaa — Z> — = — --, 
T(n) = =) n+ Toa (2n—-1)n+ 3 a0 n 3 
Thus, using the asymptotic notation introduced in Chapter 7, we have 
T(n) = Q(n’). 
(b) The recursion tree for the recurrence T(n) = T(n/3) + T(2n/3) + cn is shown in 
what follows. 
cn > cn 
S =e >cn 
F zen xn den >cn 
TA) © TA) TQ) © TA) TU) => TA) TAa) = TA) 
Since we are finding a lower bound, the height of the tree is decided by the 
shortest path, which is the leftmost path. So, the height of the tree, h, is obtained 
when n/ 3" = 1. Hence, h = log,n. From the tree, the total cost at level i is cn. 
It follows that 
log,n logn 
T(n) > 2a = cn(1 + log,n) = cn ¢ + ver) : 
Thus, using the asymptotic notation introduced in Chapter 7, we have 
T(n) = Q(n logn). 
6.1 (a) (ii). (e) (iii). (i) (i). (m) (i). 
(b) (i). (f) (iv). Q) ©. (n) (ii). 
(c) (iv). (g) (iii). (k) ©. (o) Gii). 


(d) Gi). (h) (ii). D Gv). (p) Gv). 
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6.2 


6.3 


6.4 


6.5 


6.6 


From the binomial theorem it follows that 


i ) xSokyk 


(o)a + (a) ey (2) + (3 a+ (Smt + (5) 


xX? + 5x4y + 10x*y? + 10x7y? + Sxyt + y’. 


ll 
Me 


(x+y) 


A set consisting of n elements possesses a grand total of 2” distinct subsets. Each of 
these subsets can contain zero elements, one element, two elements, and so forth, 


up to n elements. We find that there are (a) subsets that contain precisely zero 


elements, h ) subsets with one element, (3) subsets with two elements, and so 
on, until ( 7 subsets with a full complement of n elements. Hence, the summation 


> ( i ) accounts for the overall count of subsets within a set that has n elements. 


By equating the two formulas we have developed to represent the number of subsets 
within a set of n elements, we find that 


(t= 


k=0 


Using the binomial theorem with x = 2 and y = 1, we have 
n n 
youyi at = > (7) at 
(2+1) À s 2 


Consider a set T consisting of n elements. Let us select an element, say a, from this 
set, and define another set S as the complement of a within T. It is important to note 
that within the set T, there are ( K ) subsets containing exactly k elements. However, 
we can observe that a subset of T containing k elements can be in one of two forms: 
either it includes a along with k — 1 elements from S, or it comprises k elements from 


S and does not include a. Given that there are os ) subsets comprising k — 1 ele- 


ments from S, it follows that there are ( | ) subsets consisting of k elements from T 


n-1 


that include the element a. Moreover, there are ( i 


) subsets comprising k elements 
from T that do not include the element a, as as ) is the count of subsets of k ele- 


ments from S. Consequently, we arrive at the relationship: ( ) = ( ae ) + ( me i 


Consider a scenario where there are m items in one set and n items in a second set. 
The total count of ways to select r elements from the combination of these sets is 
denoted as ( we i; Alternatively, we can approach the selection of r elements from 
the union by choosing k elements from the second set and r — k elements from the 
first set, where k is an integer satisfying 0 < k < r. Since there are ( ) ways to pick 


6.7 


6.8 
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k elements from the second set and ( ne ) ways to pick r — k elements from the first 
set, we can apply the product principle of counting to determine that this can be 


accomplished in ( mY G 


See ) ways. Consequently, the total count of ways to choose r 


elements from the union is also equivalent to ),_, ( a ( $ T We have now estab- 
lished two distinct expressions for the count of ways to select r elements from the 
union of a set containing m items and another set with n items. By equating these 


expressions, we arrive at Vandermonde’s identity. 


Define the following predicate: 


P(m) = “If there are n, ways to perform task T, for k = 1, ... ,m, then there are n, x 
N, X: -X Nm ways to perform all m tasks.” 


Base case (for m = 1): P(1) is trivially true. 

Inductive step: Assume that P(k) is true for all k < m. To prove that P(m + 1) is also 
true, assume that we have m + 1 tasks, and that task T, can be performed in n, ways, 
for k = 1,2, ...,m + 1. We would like to show that the entire set of m + 1 tasks can 
be performed in n, X n, X: +: X Nm X Nyy Ways. 


Let T be a (large) task that comprises of the combination of tasks T, through T,,,. 
Now, to do the tasks T, through T,,,,,, we must perform the pair of tasks T and 
T 41. From the inductive hypothesis, we conclude that the task T can be done inn 
ways where n = n, X n, X--- X Nm ways. Now, the inductive hypothesis can be also 
used to conclude that the number of ways to perform the pair of tasks T and T,,,,; 
isn Xn,,,, ways. Thus, the entire set of m + 1 tasks can be performed in n x n 


Ny XN, X: X Ny X Nyy, Ways. 


m+1 = 


Define the following predicate: 


P(m) = “Ifa procedure can be done in one of n, ways, in one of n, ways, ..., or in one 
of n,, ways, then the total number of ways to do the procedure isn, +n, +---+N,, 
ways, provided that none of the set of n; ways of doing the procedure is the same as 
any of the set of n, ways, for all pairs i and j with 1 <i < j < m.” 


Base case (for m = 1): P(1) is trivially true. 


Inductive step: Assume that P(k) is true for all k < m. To prove that P(m + 1) is also 
true, assume that a procedure can be done in one of n; ways, in one of n, ways, ..., 
in one of n,, ways, or in one of n,,,, ways, where none of the set of n; ways of doing 
the task is the same as any of the set of n, ways, for all pairs i andj with 1 <i < j < 
m + 1. We would like to show that the total number of ways to do the procedure is 
nN, +n, +: + + Np ways. 


Note that it is assumed that the procedure can be done in one of n, ways, in one of 
n, ways, ..., in one of n,, ways. So, by the inductive hypothesis, we conclude that 
the procedure can also be done in one of n ways where n = n, +n, +-+ + nm ways. 
Note that it is also assumed that the procedure can be done in one of n,,,, ways. 
Because none of the set of n; ways of doing the procedure is the same as any of the 
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6.9 


6.10 


6.11 


6.12 


set of n; ways, for all pairs i andj with 1 <i <j < m +1, it follows that none of the 
set of n ways of doing the procedure is the same as any of the set of n,,,, ways. Now, 
the inductive hypothesis can be also used to conclude that the procedure can be 


performed inn+n,,, = n +N, +++: +N, + nmp Ways. 


Suppose, in the contrary, that none of the k boxes contains more than [N/k] — 1 
objects. Then the total number of objects would be at most 


N N 
(LE ]-1) <e((z+:) -9 = 
where we have used the inequality [N/k] < (N/k) + 1. This contradicts the fact that 
there are a total of N objects. 


Note that A'ABAC=BNCNA'=BNC-A. Then |A'ABNC|=|BnC|- 
IANBNC| =19-11=8. 


Let S be the set of all people who took the survey, T be the set of all people who like 
tea, C be the set of all people who like coffee, and M be the set of all people who 
like milk. Then |S| = 240, |T| = 91, |C| = 70, |T n C| = 31, |T’ n C’ n M’| = 91, and 
|ITACNM| =7. Weare looking for |T’ NC’ NMI. 

Note that |TUCUM| = |S|-|(TUCUM)'| = |S| — |T’ n C' n M'| = 240-91 = 
149. Note also that the subset MN T’ n C” is the dark-colored area in the Venn 
diagram in Figure A.2. It follows that 


IMNT'NC’| = |MUTUC|-|T|—|C| +|TNC| = 149 — 91 — 70 + 31 = 19. 


Note that the “for” loops in lines (5) and (8) are independent, but this pair of loops 
and “for” loops in lines (2) and (3) are nested. Let T, be the number of times of 
executing the statement given in line (s) for each s = 1,2,3,4, 5,6,8,9. The initial 
value of “sum” is one. Each time the “sum” statement in each of lines (4), (6), and 
(9) is executed, 1 is added to “sum.” Therefore, by the sum principle of counting, 
the value of “sum” after the fragment given in Algorithm 6.5 has been executed is 
equal to the number of ways to carry out the task T,, plus the number of ways to 
carry out the task T,, plus the number of ways to carry out the task T,, plus the 
number of ways to carry out the task T}. By using the product principle of counting, 


Figure A.2 A Venn diagram for Exercise 6.11. 
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we can find the number of ways to carry out the task T, s = 1,2,3,4,5,6,8,9. We 
added a column for these numbers in Algorithm A.1 (see the comments in gray in 
Algorithm A.1). Thus, the value of “sum” after the fragment given in Algorithm 6.5 
has been executed is equal to 


l+tnm+nmp+nmgq=1+nm(1+p+q). 


Algorithm A.1: Algorithm 6.5 revisited 


1: sum =1 // Task T, is done in 1 way 
2: for (i = 1;i < n;i + +) do // T, is done in n + 1 ways 
3: | for (j= 0; j< m;j+ +)do // T, is done in n x (m + 1) ways 
4: sum = sum + 1 // T, is done in n X m ways 
5: for (k = p;k > 1;k — —) do // T; is done in n x m x (p + 1) ways 
6: | sum = sum + 1 // T; is done in n X m x p ways 
7: end 
8: for (r = q;r > 0;r — —) do // T; is done in n x m x (q + 1) ways 
9: | sum = sum + 1 // T, is done in n x m x q ways 
10: end 

u: | end 

12: end 

6.13 Suppose that for each element y in the codomain of f we have a box that contains 


6.14 


6.15 


all elements x of the domain of f such that f(x) = y. Because the domain contains 
k + 1 or more elements and the codomain contains only k elements, the pigeonhole 
principle tells us that one of these boxes contains two or more elements x of the 
domain. This means that f cannot be an injection. 


Note that the order in which the shirts can be selected does not matter, and the shirts 
can be repeated. Total number of different colored shirts is n = 6. The number of 
shirts to be selected is r = 4. Here, the desired number is equal to the number of four- 
combinations with repetition allowed from a set with six elements. From Theorem 
6.13, the shirts can be displayed in 


C(6 +4- 1,4) = C(9, 4) = C(9, 5) = 126 


9-8-7-6 
1-2-3-4 
different ways. 
(a) We count permutations of 2O’s, 1 N, 1 E, 1 W, 1R, and 1 D, a total of seven 
symbols. By Theorem 6.14, the number of these is 
7! 
2!1!1!1!1!1! 
(b) Under the given condition, we should treat the O-block as a single unit “OO.” 
We then have to count permutations of 1 N, 1 E, 1 W, 1 R, 1D, and 1 “OO,” a total 
of six symbols. The number of these is 
6! 


=7-6-5°4-3 = 2,520. 


= 6! = 720. 
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6.16 (a) Wecount permutations of 3 P’s, 2 E’s, 1 I, 1 N, 1 A, and 1 L, a total of nine symbols. 


By Theorem 6.14, the number of these is 
9! 
3!2!1!1!1!1! 
(b) Under the given condition, we should treat the P-block as a single unit “PPP.” 
We then have to count permutations of 2 E’s, 1 I, 1N, 1 A, 1 L, and 1 “PPP” a 
total of eight symbols. The number of these is 
8! 


= 30, 240. 


ammm e1 
7.14 (a) (iii). (g) (ii). (m) (ii). (s) (ii). 
(b) 6). (h) (iv). (n) (iii). (t) (ii). 
(c) ©. © ii). (0) (iv). (u) (iv). 
(d) (i). (j) (iv). (p) (ii). (v) (iv). 
(e) (i). (k) (iv). (q) (iii). 
(f) Gi). (1) (iv). (r) (iv). 


7.2 An algorithmic code that solves the given problem is stated in Algorithm A.2. 


Algorithm A.2: The algorithm of Exercise 7.2 


Input: An array A[0 : n — 1] of positive integers and length n 
Output: The largest number of the array 


: find-maxi(A,n) 
: if (n = 1) then 

i | return A[0] 

: end 

: v1 = A[0] 

: V2 = find-maxi(A,n — 1) 
: if (v1 > v2) then 
: | return v 

: end 

: else 

: | return v, 


end 


7.3 (a) Yes. The input for Algorithm A.2 is a finite array of positive numbers whose 


length is n. 
(b) Yes. The output for Algorithm A.2 is the maximum in the array. 
(c) Yes. Algorithm A.2 terminates when n is equal to 1. 
(d) Yes. One can prove the following equality by mathematical induction. 
find-maxi(A[i : n — 1],n — i) 
= max {A[i], find-maxi(A[i+ 1 : n — 1], n — i — 1)}. 
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7.4 Let T(n) be the running time of Algorithm 7.12. Carrying out a line-by-line analysis 
for Algorithm 7.12, we have 


Line One-time cost Number of times 
2 Cy 1 

3 Cy n+1 

4 (o n 

5 C, n [worst-case] 

8 c 1 


The total runtime is 


T(n) = & +0,(n+1)+¢,n + Cyn +c, = (Cy +c, + C4) + (c; + Cy +c). 


7.5 Let T(n) be the running time of Algorithm 7.33. Carrying out a line-by-line analysis 
for Algorithm 7.33, we have 


Line One-time cost Number of times 
1 Cy 1 

2 c 1+logn 

3 Č; log n(1 + logn) 

4 Cy log’n 


The total runtime is 
T(n) = c, +¢,(1 + logn) +c, log n(1 + logn) + c,log’n 
= (c, +c,)log’n + (c, + c,)logn +c, +c). 


7.6 (a) Since the loop goes from p to q in increments of 1, and p < q, the solution would 
be q — p. Since this loop includes q as part of the end condition (< q), add one to 
the solution, making it q — p + 1. 

(b) Since the loop goes from q to p in decrements of 1, and p < q, the solution would 
be q — p. Since this loop includes p as part of the end condition (> p), add one to 
the solution, making it q — p + 1. 

(c) Since the loop goes from p to q in increments of k, and p < q, the solution would 
be [(q — p)/k] + 1. Since the loop includes q as part of the end condition (< q), 
add one to the solution, making it |(q — p)/k] + 2. 

(d) Since the loop goes from q to p in decrements of k, and p < q, the solution would 
be [(q — p)/k] + 1. Since the loop includes p as part of the end condition (> p), 
add one to the solution, making it |(q — p)/k] + 2. 
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7.7 


7.8 


7.9 


7.10 


(e) The loop goes from p to q as follows: p,kp,k?p,kĉp,...,q—1,q. Note that 


k"p = q when h = log,(q/p). So, the solution would be log,(p/q)+ 1. Since 
the loop includes q as part of the end condition (<q), add one to the solution, 
making it log,(p/q) + 2. 

(£) The loop goes from q to p as follows: q, g/k,q/k’. q/k*, ...,p +1,p. Note that 
q/k" = p when h = log,(q/p). So, the solution would be log, (p/q) + 1. Since the 
loop includes p as part of the end condition (> p), add one to the solution, making 


it log,(p/q) + 2. 


Let T(n) be the running time of Algorithm 7.34. Writing the summations that rep- 
resent the running time of the code in Algorithm 7.34 and solving the summations, 
we get 


n?+5 n n?+5 
T(n) = DY } a= } am- 3) = cn? +5- n? +1- 3) = 6c,(n — 3). 
i=n? j=4 i=n? 


Let T(n) be the running time of Algorithm 7.35. Writing the summations that rep- 
resent the running time of Algorithm 7.35 and solving them by bounding, we get 


3i 
T(n) = SSe- = 5o c= a < se = = 3cnî. 
i=1 j=1 i=1 


Thus, T(n) = O(n’). 


For an upper bound, we have 


4n? 8n? 4n? 4n? 


f=) Ye= Len’ -i+ Des Der" Je < (4n? = n + 1)(8n°)c. 


i=n j=i i=n 


It follows that f(n) < (32n® — 8n* + 8n3)c, and therefore f(n) € O(n). 
For a lower bound, we have 


4n? 8n? 4n? 4n? 
fw = VY Ye= X Er - de = X 8n - 4n’) > (4n? -n + 1)(4n*)e. 
i=n j=i i=n i=n 


It follows that f(n) > (16n° — 4n* + 4n*)c, and therefore f(n) € Q(n°). 
Thus, from Property 7.1, we have f(n) € (n°). 


From Algorithm 7.30, we have 


n=l n j n-l1 n 
f(n) = ¥ YY See Y Vania 
i=] j=i+1 k=i i=1 j=i+1 


For an upper bound, we have 


n-i n n-l n 


fom se, Jis) Znas eYin— on se = cn?(n — 1) < enè. 


i=1 j=i+1 i=1 j=i+1 
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For a lower bound, we have 


n-i n 


fm >cY YG-d 


i=1 j=i+1 


n=l n 
cy DY g- 
i=1 j=(n—i)/2 
n-i n 
22 
i=1j=(n—i)/2 


IV 


IV 

| z 
| 

| 

~. 


n-1 


n 
= n =) 
Be, ( 2 


IV 
M 
~~ 
NIS 
| 
NI 
WS 
n 
| 

= 
N| | 
~. 


IV 
° 
~ 
NIS 
| 
NIS 
x 


IV 

° 
7 5 
ll 
= N 
AT 
AJS, 

| 
A/S 


IV 
Qa 
= 
Il 
= N 
AT 
|= 
| 
I= 
fo 
NIS 
— 
N 
See ae 


Thus Sen} < f(n) < cn’, and therefore f(n) = @(n?). 


7.11 (a) Ifn > 1, then 
5n? — 3n + 20 < 5n? + 20 < 5n? + 20n? = 25n’. 


Therefore, 5n? — 3n + 20 € O(n?) with c = 25 and ny = 1. 
(b) Ifn > 1, then 


4n? — 12n + 10 < 4n? + 10 < 4n? + 10n? = 14n?. 


Therefore, 4n? — 12n + 10 € O(n”) with c = 14 and n, = 1. 
(c) Ifn > 1, then 


5n? —4n* — 2n? + n < 5n? +n < 5n? +n? = 6n?. 


Therefore, 5n° — 4n* — 2n? + n € O(n’) with c = 6 and n, = 1. 
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7.12 


7.13 


7.14 


(d) Ifn > 2, then 
n? + ynsin(n) + nlog(n) < n? + (n(n) + (nX(n) < 3n?. 
Therefore, n? + ynsin(n) + nlog(n) € O(n?) with c = 3 and ny = 2. 
(a) Ifn > 1, then 4n? +n + 1 > 4n?. Therefore, 4n? + n + 1 € Q(n?) with c = 4 and 
n=l. 
(b) If n > 1, then nlog(n) —2n + 13 > nlog(n)— 2n. It is enough to find pos- 
itive constants c and ny such that nlog(n) — 2n > cnlog(n), or equivalently 
1 — 2/log(n) > c, forall n > ng. Note that, for n > 8, we have 1 — 2/ log(n) > 1/3. 
Therefore, n log(n) — 2n + 13 = Q(n log(n)) with c = 1/3 and ny = 8. 
(a) Ifn > 1, then 
n? <n? +R Tnn? n? HT +n = On. 


Therefore, n> + n? + 7n + 1 € O(n°) with c) = 1, c, = 10 and ng = 1. 
(b) Ifn > 7, then 


Lc ly 3n < 1n?, 
14 2 2 
where the first inequality follows by noting that 
112 —3n> anr a be? >c. Hence, for n > 7,4 - Ž > Ł =e. 
2 2 n 2 n 14 


Therefore, in? -3n = O(n’) with c, = 1/14, c, = 1/2 and ny =7. 


Given that f,(n) = O(g,(n)) for each i = 1,2,...,k, then by definition there are posi- 
tive constants c, and n; such that f(n) < ¢,g,(n) for n > nj. 
(a) Let ng = max j<<,N; and Cy = C} +C, +--+ + Cp then for all n > ng, we have 


k i A 

Dil) < Lies) < (Ys) max g;(7) < comaxg;(n). 
Thus, Efn = O(max ;<i<&:(n)). 

(b) Let ny = max ,<;,M; and cy = c,C, +++ Cp then for all n > ng, we have 


k k k k k 
ke) < [sim = (IIs) [flew = co [gi(n). 
i=1 i=1 i=1 i=1 i=1 

Thus, Maf = oJI gn)). 


We prove Properties 7.1 and 7.6, and prove also Properties 7.2 and 7.7 for Big-O. 

Property 7.1 Pf. By definitions, f(n) = O(g(n)) and f(n) = Q(g(n)) iff there are posi- 
tive constants c,,c,,n, and n, such that f(n) < c,g(n)forn > n,,and 
that f(n) > c,g(n) for n > n,. This is equivalent to c,g(n) < f(n) < 
c,g(n) for all n > max {n,,n,}, which is also equivalent to f(n) = 
A(g(n)), with ng = max {n,,n,}, by definition. Thus, we conclude 


f) = ogn) iff) = O(g(n)) and fin) = Q(g(n)). 


7.16 


7.17 
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Property 7.2 Pf. Given that f(n) = O(g(n)), then by definition there are positive con- 
stants c} and n, such that f(n) < cg(n) for n > n,. Similarly, given 
that g(n) = O(h(n)), then by definition there are positive constants 
c, and n, such that g(n) < c,h(n) for n > n,. Let ny = max {n}, n3}, 
then f(n) < c,g(n) < c,c,h(n) for all n > ng. Let cy = cic, then 
f(n) <c h(n) for all n > ng, which means we can conclude that 
f(n) = O(n). 

Property 7.6 Pf. By definition of Big-Oh, f(n) = O(g(n)) iff there are positive con- 
stants cand n, such that f(n) < c,g(n), or equivalently g(n) > : f(n), 
for all n > no. By definition of Big-Omega, this equivalent to g(n) = 
Q(f(n)) with positive constants 1/c and np. 

Property 7.7 Pf. Note that f(n)<f(n) for n>1. Thus, f(n)=O(f(n)) with 
c=ny=1. 


(a) We have y4n? + 1 = O(n) because 


V4n24+1 2 
fim tg OE elima 4 aa A te 
n>% n n>% n2 n>o n2 noo n2 


(b) We have n” = Q(n!) because 


n-times 
—— 
. n” . n-n---n-n A n n n 
lim = = limet = lim (1, . Eon) =o. 
noon! noon-(N—1)---2-1 n>% n-1 n-2 2 


(c) We have yn +4- yn = O(1) because 


n> co 1 n>% Vn+4+ yn 
; (n+4)-n 
= lim 
n> In+4+/n 
= lim 4 = 


(d) We have n log(n?) + (n — 1)” log(5) = O(n? log(n)) because 


nlog(n) + (n — 1} log(*) ties 2nlogn + (n — 1)?(logn — log 2) 


li 
nen n? logn n=% n? logn 
— lim 2 P (n — 1}? (logn — log 2) 
n>% n n2 log n 


Yes, it is true that 2”*'® = O(2”). To prove this, note that, if n > 1, then 2”™® = 
2102" < (1024)(2”). Thus, 27?” = O(2") with c = 1024 and ny = 1. 
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7.18 


7.19 


7.20 


(a) dii). (b) 6. (c) (ii). (d) 6). 


The first statement is an assignment statement with constant time. The inside of the 

for loop is also a simple statement with constant time. So, the part that will affect 

overall runtime is the runtime and value of f(n). The answers are (a) O(n?) and (b) 

O(n(n!)). Below is the reasoning behind these answers, which is similar to that rea- 

soning of Example 7.36 (b). 

(a) The“test” runtime in the for loop is O(n), since that is how long it takes f(n) to 
run. The “body” of the for loop takes O(1), as it is just a simple statement. The 
“reinitialization” runtime still takes O(1). We perform the loop O(n!) times, since 
the value of f(n) is n. Thus, putting it all together, we have O(1 + (n + 1 + 1)n) = 
O(n’). 

(b) The“test” runtime in the for loop is O(n), since that is how long it takes f(n) to 
run. The “body” of the for loop takes O(1), as it is just a simple statement. The 
“reinitialization” runtime still takes O(1). We perform the loop O(n!) times, since 
the value of f(n) is n!. Thus, putting it all together, we have O(1 + (n + 1 + 1)n!) = 
O(nn!). 


The answer is O(n?). Below is the reasoning behind the answer. 

In Figure 7.9, we see how the runtime of the loops is calculated. We add the initializa- 
tion runtime (usually O(1)) plus the cost of going around the loop once multiplied 
by the number of times we go around the loop, represented O(1 + (f(n) + Dg(n)). 
Keep in mind “the cost of going around the loop once” is represented in Figure 7.9 
as the runtimes of “test” plus “body” plus “reinitialize,” with “test” being the condi- 
tion of the for loop and “body” obviously being the body. Now, we apply this to our 
problem. 

The “test” runtime in the for loop is O(n), since that is how long it takes the function 
cat (.) torun and, as previously found, cat (.) has a runtime of O(n). The “body” 
of the for loop takes O(n), since, again, that is how long it takes cat (.) to run. The 
“reinitialization” runtime takes O(1). 

The “for” loop in line (10) goes from 1 to the value of the function cat (n,n).Thus, 
we must find what is the value of cat (n,n). The function cat (.) takes a number 
and adds values from 1 to n to the original number, in this case n. So, the value of 
cat (n,n) equalsn+1+2+3+4---+n. In other words, 


n(n+1)_ n*+3n 


€ O(n’). 
7 5 (n^) 


n 
cat(n,n) =n + Ñi= n+ 
i=1 


Thus, the for loop is iterated O(n?) times (from i = 1 toi = (n? + 3n)/2). Now, putting 
it all together we have O(1 + (n + n + 1)n?) = O(n’). 

The first two statements of the function main (.) are O(1), as they are simple state- 
ments. The statement in line (3) calls the function cow ( . ) , which takes O(n?) as we 
found above. The runtime of the statement in line (4), which has a call to cat (.), 
is O(n) (as previously stated, cat (.) has a runtime of O(n)). So, the total runtime 
for main(.) is O(1) + O(1) + O(m?) + O(n) = O(n). 


Begin algorithm 


initialize e, y, 0, 6, x, u®, A®, f,A 


M 


x£x0,n 4 uO, AAO 


miu 2 €? 


y 
m~~ [Isk< K+1? 


y 


no 
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7.21 The graph structure of Algorithm 7.38 is shown in Figure A.3. 


yes y 
k=1 


no 


find (y®, 2), s®) 
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compute g 
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compute (ys A s) 
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k++ 


aaa 


-——————> | Is 6 > f? 


Y 
compute AA 
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compute ô(u, x) 


Y 


no 


m~~ |Isk< K+1? 


| 
k=1 
| 


no 
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k) (© 
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find (y®, 2®, s®) 


y 


y 
compute g 


Yy 


choose a scaling element p 


compute Ax 


y 


compute (as, a) 


y 


k++ 
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v 
compute AA 


Y 


Y 


compute ô(u, x) 


compute u = yu 


Figure A.3 The graph structure of Algorithm 7.38. 
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End algorithm 
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7.22 The running time of the block in lines (17)-(19) is O(m), so the running time of the 


for-statement in lines (16)-(20) is O(Km). Since the running time of the block in 
lines (21)-(24) is O(n) and those in the statements in lines (14) and (15) are O(1), the 
running time of the block in lines (14)-(25) is O(n + Km), and hence the running 
time of the while-statement in lines (13)-(25) is O(n + Km)N,,,). Here, N,„ is the 
number of times we go around the inner while loop of line (13). The running time 
of the block in lines (5)-(7) is O(m), so the running time of the for-statement in lines 
(4)-(8) is O(Km). Since the running time of the block in lines (9)-(12) is O(n), the 
running time of the block in lines (4)-(12) is O(n + Km), and hence the running time 
of the block in lines (4)-(26) is O(n + Km) + (n+ Km)N,,,) = O(n + Km)N,,), it fol- 
lows that the while-statement in lines (3)-(27) is O(n + Km)N,,,N,,,), Where Nut is 
the number of times we go around the outer while loop of line (3). As a result, based 
on the information obtained and because the running time of the for-statement in 
lines (28)-(30) is O(Km), and all other statements are of constant times O(1), the 
running time of the algorithm is 


O + Km + (n+ KMN Nou) = O(n + KMN Nout): 


Bounding N,,, is exactly similar to bounding Nut in Example 7.35. Using a similar 
argument to that in Example 7.35, we can show that 


log(u /e) 
-logy ` 
(a) It is given that N,, = O(n + Km). Now, if y € (0,1) is an arbitrarily chosen con- 
stant, then y = O(1), and hence 


N, 


out = 


u® 
Nout < log (2) o(1). 
€ 


Thus, the running time of the algorithm as a whole is 


pO 
O(n + KMN pa Nout) = O (o + Km» log ( ) ) l 


e 
(b) It is given that N;, = O(1). Now, ify = 1 — o/y n + Km (o > 0), then! 
logy = log (1 — o/yn+Km) x —o/yn+Km, 


and hence 


log(u® logu® © 
e a M L TE , Vnt Kmlog( £) OU). 
-logy o/y n + Km € 


Thus, the running time of the algorithm as a whole is 


(0) 
O(n + KMN Nou) =0 (o + Km)?” log (<)) . 
E€ 


1 For small positive values of x, we have log(1 + x) x (1 +x)- 1 =x. 


8.1 


8.2 


8.3 


8.4 


8.5 
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(a) (iii). (d) @. (g) (iv). 
(b) (iv). (e) (iii). (h) (iii). 
(c) (ii). (f) (ii). (i) (ii). 


When we write O(n?376), we mean that there is a positive constant c such that the 
Coppersmith and Winograd’s algorithm takes no more than cn?” flops. For this 
algorithm, the constant c is so large that it does not beat Strassen’s method until n is 
really enormous. 


The program in Algorithm 8.14 is nonrecursive. Figure A.4 shows a basic scheme of 
this program. From Algorithm 8.5, the function 1inear-search is O(n). In view 
of Figure A.4, we analyze the function karger before analyzing the function main 
because the later calls the function karger. Now, if the function call in the body of 
a for loop, we add its cost to the bound on the time for each iteration. It follows that 
the running time ofa call to karger is O(mn). Next, when the function call is within 
a simple statement, we add its cost to the cost of that statement. Thus, the function 
main takes O(mn) times. Therefore, the running time of this program is O(mn). 


The program in Algorithm 8.15 is nonrecursive. Figure A.5 shows a basic scheme 
of this program. From Algorithms 8.6 and 8.9, the functions 1inear-search and 
merge-sort are O(logn) and O(nlogn), respectively. In view of Figure A.5, we 
analyze the function dinic before analyzing the function main because the later 
calls the first. Now, if the function call in the body of a for loop, we add its cost 
to the bound on the time for each iteration. It follows that the running time of 
a call to dinic is O(log mlogn). Next, when the function call is within a simple 
statement, we add its cost to the cost of that statement. Thus, the function main 
takes O(log m logn +n logn) times. Therefore, the running time of this program is 
O((n + logm) logn). 


If x divides both a and b, then it clearly also divides a — b|a/b| =a mod b. 
Conversely, if x divides a mod b and b, then it also divides (a mod b) + b|a/b| = a. 


function karger calls the function linear-search. 
We already analyzed the function linear-search in 
Algorithm 8.5. Thus, we analyze the function karger, 


The function main calls the function karger, and the (main 


and then analyze the function main. 
arger 


linear-search 


Figure A.4 A basic scheme of the program shown in Algorithm 8.14. 
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The function main calls the functions (main 
dinic and merge-sort, and the 
function dinic calls thef unction 
binary-search. We already ana- 
lyzed the functions binary-search © 
and merge-sort in Algorithms 8.6 


and 8.9, respectively. So, we analyze 
the function dinic, and then analyze 


the function main. 
merge-sort 


Figure A.5 A basic scheme of the program shown in Algorithm 8.15. 


8.6 We need to show that ifa > b > 0 and Algorithm 8.10 performs n > 1 iterations of 
the while loop, then a > f,,,, and b > fagi- 
The proof is by induction on n. The statement holds for n = 1 because b > 1 = fy 
anda > bimply thata > 2 = f}. Now assume that n > 2. Because a > b > 1andn > 
2, the next iteration will use the number b and a mod b. To these, we can apply 
the induction hypothesis, because n — 1 iterations remain and b > a modb > 0 (as 
n-— 1 > 0). Thus, b>f,,, and amodb > f,. Thus, a = |a/b] -b+(amodb) > b + 
(amod b) > frst + fa = fanz 


8.7 This is an implementation exercise. Applying Newton’s method, we obtain the root 
3.15145 if we start with x, = 1. 


9.1 (a) (iv). (d) (ii). (g) (iv). (j) (iii). 
(b) (ii). (e) (iii). (h) (iv). 
(c) (iii). ®© ©. (i) (ii). 

9.2 (a) True. (d) False. (g) True. (j) False. 
(b) True. (e) True. (h) True. 
(c) True. (f) False. (i) False. 


9.3 (a) The adjacency list representation for the graph is: 


o0 N (oN nn e U N 
e 
Y 
Ww w nN N N N 
& 
Y 
U 
(J 
Y 
lon 
v 
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(b) The adjacency matrix representation for the graph is: 


male lolelofoleJelw 
oļoļololoļoļ=lo]> 
oļloļl=loloļ=ļ=lolļu 
olļloļlolļl=lolololola 


oo e = p= => m= |w 
= jojojo joj=e |oo] 
Oy J|/olol]o}]RK|o;o]|o 


SC}l/O;JO;SO;SOlRe [RF l/o]teR 


9.4 (a) Running a breadth-first search, we obtain the breadth-first tree shown to the 
right. It follows that (v,, v5, Us, Ug, Ug) is the shortest path from vertex v, to vertex 
vg. This shortest path is clearly unique. 


(b) As shown below, running a breadth-first search, we find that there is a conflict- 
ing assignment which occurs when we color the vertex v}. Hence this is not a 
bipartite graph. 


(i) (ii) (iil) 
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(c) As shown below, running a depth-first search, we get the following depth-first 
tree. 


(iii) 


(i) (ii) 


(iv) (v) (vi) 


In (viii), the solid edges 
are the tree edges, and 
dotted edges are the cross 
edges. 


(vii) (viii) 


(d) Because we did not find a back edge we performed a depth-first search in item 
(c), the directed graph has no a directed cycle. 
(e) A topological ordering is computed and shown below. 


9.5 (a) Running a breadth-first search on the graph, we obtain the following breadth- 


first tree. 
©) GX} 9) 
o TA 
© C9 
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(b) Running a breadth-first search on the given graph, we find that the vertices s and 
r have the same color as shown below. Thus, this is not a bipartite graph. 


HAAA 


(c) Running a depth-first search on the graph, we obtain the following depth-first 
tree. 


© 9 
©) OA 
Q Q 


(d) Because we did not find a back edge we performed a depth-first search in item 
(c), the directed graph has no a directed cycle. 

(e) See the solution of item (e) in Exercise 9.4. This graph and that in Exercise 9.4 
are isomorphic, and they have the same topological ordering. 


9.6 (a) Running a breadth-first search on the graph, we obtain the following breadth- 
first tree. 


(b) The undirected version of the graph is bipartite as shown below. 
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(c) Running a depth-first search on the graph, we obtain the following depth-first 


O—-) &)--@) 
O Oa 
O-O O-&) 


(d) Because we did not find a back edge we performed a depth-first search in item 
(c), the directed graph has no a directed cycle. 
(e) A topological ordering is computed and shown below. 


6-0-6 Oo o> 


9.7 This is a false statement. The depth-first search may produce different depth-first 
trees with different numbers of tree edges depending on the starting vertex and upon 
the order in which vertices are searched. 

As a counterexample, consider the graph shown to the right. If the depth- © 
first search starts at u, then it will visit v next, and (u, v) will become a tree 
edge. But if the depth-first search starts at v, then u and v become separate 
trees in the depth-first forest, and (u, v) becomes a cross edge. C) 
10.1 (a) (ii). (8) ©). (m) (i). (s) (v). 
(b) (ii). (h) (i). (n) v). (t) (ii). 
(c) Gi). (i) 6). (o) (ii). (u) (ii). 
(d) Gi). Q) Giù. (p) ©. 
(e) v). (k) (ii). (q) @. 
(f) (iv). D (). (r) ii. 
10.2 The decision variables are: 


x, = kg of food F}; 
x, = kg of food F,. 
Minimizing the cost of the mixtures, we have the following LP problem. 


min 60x, + 80x, 

st. 5x,+ 2x, 211, 
3x, + 4x, = 8, 
X; X% 20. 
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10.3 The decision variables are: 


xı: The number of loaves of bread baked; 


x: The change in the supply of flour through financial transactions (in ounces). 
Measuring profits in cents, we have the following LP problem. 


max 30x, — 4x, 

st. Sx; — X < 30, 
X <5, 
X > 0. 


10.4 The decision variables are: 


x,: Acres of radishes produced; 
x,: Acres of onions produced; 


xz: Acres of potatoes produced. 


We wish to maximize the agriculturist’s profit (in $). Subtracting labor and water 
costs, we obtain the following LP problem. 


max 7072.5x, + 6775x, + 4812.5x, 


s.t. xX, + X, + x3 < 126, 
6x, + 5x, + 6x; < 500, 
Xis Xz, x,2 0. 

A x, 


1.25x1 + 5x» < 90 


150 + 


25x, + x2 < 65 


xX; + x2 < 85 
15x, + 10x. = 0 SS... 


-20 ~oo 20 40 


Figure A.6 Graphical solution of the optimization problem in Exercise 10.5. 
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10.5 The graphical representation of the given LP problem is shown in Figure A.6, with 
the feasible region shaded in cyan. From the graph, we find that the minimum 
value for z is 0 at x = (0, 0). 


10.6 (a) The decision variables are: x = the number of tons of product X, and y = the 


number of tons of product Y. The LP problem that can be used to maximize the 
profit is seen below: 


max 80x + 120y 


s.t. x > 2, 
y= 3, 

x+ y= 9, 
20x + 50y < 360, 

X, y2 0. 


(b) The feasible region is the thick black line segment shown in the graph in 
Figure A.7. We find that the maximum value for the objective function is 960 
at x = (3, 6). 


10.7 (a) Letx, be the number of tons produced by paint P, and x, be the number of tons 
produced by paint P,. We have the following LP: 


10 + 


80x + 120y = 960 ~.. 


20x + 50y < 360 


es 
IV 
N 


—2 + 


Figure A.7 Graphical solution of the optimization problem in Exercise 10.6. 
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max 500x, + 400x, 
st. 6X, + 4x, 
x+ 2x, <6, 


500x; + 400x2 = 2000 ~ 


x] 


6x, $ 4x2 < 24 


xı + 2x. < 6 


Figure A.9 Graphical solution of the optimization problem in Exercise 10.7 (c). 
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500x; + 400x2 = 1800 ~. 
6x1 + 4x2 < 24 xj- 42S 1 


Figure A.10 Graphical solution of the optimization problem in Exercise 10.7 (d). 


(b) The graphical representation of the given LP problem is shown in Figure A.8, 
with the feasible region shaded in cyan. From the graph, we find that the max- 
imum value for the objective function is 2000 at x = (8/3, 5/3). 

(c) Restricting x, to be integer-valued changes the feasible region, we find that 
the feasible region is the thick black line segments shown in the graph in 
Figure A.9. We also find that the optimal solution is now 1800 at x = (2, 2). 

(d) Restricting both x, and x, to be integer-valued again changes the feasible region 
to become the black bullets shown in the graph in Figure A.10. The optimal 
solution remains the same. 


10.8 (a) The graphical representation of the given LP problem is shown in Figure A.11, 

with the feasible region shaded in cyan. From the graph, we find that the min- 
imum value for the objective function is 230/13 at x = (18/13, 20/13). 

(b) The feasible region is the thick black line segment shown in the graph in 
Figure A.12. From the graph, we find that the maximum value for the objective 
function is 18 at x = (2, 2). 

(c) The feasible region is the black bullet shown in the graph in Figure A.13. From 
the graph, we find that the maximum value for the objective function is 3 at 
x = (2,3). 


10.9 (a) Restricting x, to be integer-valued changes the feasible region to become the 
thick black line segments shown in the graph in Figure A.14. We find that the 
optimal solution is 62/3 at x = (10/3, 1). 
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Figure A.12 Graphical solution of the optimization problem in Exercise 10.8 (b). 
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3x+2y < 12 


5x1 + 4x2 = 62/3 A 


3x, + 2x. < 12 


Figure A.14 Graphical solution of the optimization problem in Exercise 10.9 (a). 
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-xi +x < 1 


5xı + 4x2 = 20s. 


3x, + 2x. < 12 


Figure A.15 Graphical solution of the optimization problem in Exercise 10.9 (b). 


(b) Restricting both x, and x, to be integer-valued again changes to become the 
black bullets shown in the graph in Figure A.15. We find that the optimal solu- 
tion is 20 at x = (4,0). 


10.10 Letting x; = k -xXx for i = 1,2,3, and introducing the excess variable x, and the 
slack variables x, and x,, the given LP problem is equivalent to the standard form 
problem: 


min z= 2x] — 2x- 4x,'+ 4x>+ 5xj — 5x; — 30 


s.t. Say — 3x] + 2x5 — 2x,- x + 3 -xX = 10, 
—2xf + 2x7 + 4xt — 4x; + Xz = 35, 
Axi — 4x0 - xP+ xy + Xş = 20, 
=> i +x, =6, 
xy - xX; +X, = 8, 
3- g +X, = 10, 
Ais Xie x}; x5; Xs Xz, Xy, Xss XGoXqoXgiXqy 20. 


10.11 We need to show that if (DILP) is infeasible, then (PILP) is either infeasible or 
unbounded. The possibility that Problems (PILP) and (DILP) could be both infeasi- 
ble has been grounded in Example 10.38. So, to prove the desired result, it remains 
to show that if (DILP) is infeasible and (PILP) is feasible, then (PILP) must be 
unbounded. Assume that (DILP) is infeasible and let x be a feasible solution for 
(PILP). Due to the infeasibility of (DILP), there does not exist y satisfying A'y < c. 
Using Farkas’ lemma (Version II); see Theorem 3.16, there is a vector £ satisfying 
AX = 0, c'X < 0, and £ > 0. Due to the feasibility of x in (PILP), we have Ax = b 
and ¥ > 0. Define x, =X + aX for a > 0. Then 


Ax, = A(X + aX) = Ax + aAX = b+ «A$ = b, and x, =x+ax>0. 
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10.12 


10.13 


10.14 


This means that x, is feasible in (PILP). Note that, because c'X < 0, we have 
e'x, = c' (+ ak) = c'X + ac! —— cY — œ = —o, 
as a ——> oo, which implies that Problem (PILP) is unbounded. 


(x,,X5) = (4, —1/3). 


(a) (ii). (c) Gv). (e) (iii). (g) (iii). 
(b) @. (d) Gi). ®© (ii). 


(a) After introducing slack variables, say x, and x,, we obtain the following stan- 
dard form problem: 


max Z=X,+1.5x, 

s.t. 2x, + 4x, +x; = 12, 
3X, + 2x, +X, = 10, 
X1; X2; X3, X4 Z 0. 


Note that x = (0,0, 12, 10) is a basic feasible solution. Hence, we have the fol- 
lowing initial tableau: 


rhs|x,; X, X; X4 
0/1 15 0 0 
x,=|12/2 ®© 1 0 
x,=|10)3 2 01 


Since we are maximizing the objective function, we select a nonbasic variable 
with the greatest positive reduced cost to be the one that enters the basis. Indi- 
cating the pivot element with a circled number, we obtain the following two 


tableaux: 
rhs} x, xX Xz Xi rhs|x, X, X3 X4 
-9/2|1/4 0 -1/4 0 -20| 0 0 —3/4 0 
x,=| 3/1/72 1 1/4 0 x,=| 2/0 1 3/8 -1/2 
x=] 4| @ 0 -1/2 1 x =| 2|1 0 -1/4 1 


The reduced costs in the zeroth row of the tableau are all nonpositive, so the 
current basic feasible solution is optimal. In terms of the original variables x, 
and x,, this solution is x = (2, 2). 
(b) After introducing the slack variables, s,, s, and s}, we obtain the following 
standard form problem: 
max Z= 3x, + 5x, + 4x; 
s.t. 2x, + 3x, +5, = 8, 
2x, + 5x; + s, = 10, 
3x, + 2x, + 4x3 + 83 = 15, 
X X2 X3 > 0. 
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We then obtain the following sequence of tableaux when using the circled pivot 
point. 


X X, X} Sı S, S;|rhs 
2 © 010 0 8 


0 2 5 0 1 Of 10 
3 2 4 0 0 1) 15 
-3 -5 -4 0 0 0 0 
X, X, X% Sı S3 S| rhs 
2/3 1 0 1/3 0 O| 8/3 
-4/3 0 © -2/3 1 0/14/3 
5/3 0 4 -2/3 0 1|29/3 
1/3 0 -4 5/3 0 0140/3 
X, X% X% Sy S2 83 rhs 
2/3 1 0 1⁄3 0 ol 8/3 
-4/15 0 1 -2/15 1/5 0| 14/15 
0 0 -2/15 -4/5 1| 98/15 
-11/15 0 0 17/15 4/5 0256/15 
E. Si Sy S; rhs 
0 1 0 15/41 8/41 —10/41] 50/41 
0 0 1 -6/41 5/41 4/41] 62/41 
1 0 0 -2/41 -12/41 -15/41) 89/41 
0 0 0 45/41 24/41 11/41|765/41 


The reduced costs in the zeroth row of the tableau are all positive, so the basic 
feasible solution is optimal. In terms of the original variables, x,,x,, and x, this 
solutions is x = (89/41, 50/41, 62/41). 

(c) After introducing the slack variables, say x,,x;, and x,, we obtain the following 
standard form problem: 


max 2x,—- X,+ X% 


s.t. 3x, + X + X3+Xy = 6, 
Xx — X% + 2x; +X; =], 
4+- Xz +X =2, 


Xs X» Xz, Xy, Xs, Xe Z0. 


Note that x = (0,0,0,6, 1, 2) is a basic feasible solution. Hence, we have the 
following initial tableau: 


rhs|x,; X, X% X4 X; Xe 


0/2 -1 1 0 0 0 
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Since we are maximizing the objective function, we choose a nonbasic vari- 
able with the greatest positive reduced cost to be the one that enters the basis. 
Indicating the pivot variable with a circled number, we obtain the following 


tableaux: 

rhs|x xX, X} X4 X; Xe 

-2| 0 1 -3 0 -2 0 

x, =| 3/0 -5 1-3 0 

x% =| 1/1 -1 2 0 1 0 

Xe = 0 @ -3 0-1 1 
rhs|x, x X% X, Xs X6 
—2.5}0 0 -1.5 0 -1.5 —0.5 
X4 = 1/0 0 1 i =l á =2 
x% =| 15/1 0 05 0 05 0.5 
xX% =| 0.50 1 -15 0 —0.5 0.5 


The reduced costs in the zeroth row of the tableau are all nonpositive, so the 
current basic feasible solution is optimal. In terms of the original variables 
xX, and x, this solution is x = (1.5, 0.5, 0). 

(d) After introducing slack variables, say x4, X5, Xç, and x,, we obtain the following 
standard form problem: 


max Z = 60x, + 30x, + 20x, 


s.t. 8X, + 6X, + Xx+ X4 = 48, 
4x, + 2x, + 1.5x, + Xs = 20, 

2x, + 1.5x, + 0.5x3 + + X6 = 8, 

X +X, =5, 

Xj; Xp, X3, Xy, Xs, X%, X 20. 


Note that x = (0,0, 0, 48, 20, 8, 5) is a basic feasible solution. Hence, we have the 
following initial tableau: 


ths|x, X, X X4 X; Xs Xz 
0/60 30 20 0 0 0 O 
x= 488 6 1 1 
x; =| 20| 4 215 0 
xs=| 8|© 15 05 0 
x,=| 550 1 0 0 


OmoOoo 
= O O G 


0 
1 
0 
0 


Again indicating the pivot element with a circled number, we obtain the fol- 
lowing tableaux: 


rhs|x, xX, xX, 

—240| 0 —15 5 

X4 160 0 -1 
x;=| 4/0 -1 
x, = 1 0.75 0.25 
=| 50 1 0 
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rhs|x,  X, X3 X Xs Xs X% 
—280|0 -5 0 0 -10 -10 0 

X4 = 24/0 -2 0 1 2 -8 0 
x=] 8/0 -2 10 2 -4 0 
x, = 2|1 125 0 0 -05 15 0 
xX, = 5| 0 1 0 0 0 O 1 


The reduced costs in the zeroth row of the tableau are all nonpositive, so the 
current basic feasible solution is optimal. In terms of the original variables x,, 


X,, and x, this solution is x = (2, 0, 8). 


10.15 (a) We have the following tableau: 


X X Sı S, $ |rhs 
0 0 17 -7 0|10 
1 0 3 -1 Oj 2 
0O 1 4 -2 O| 2 
0 O 1 0 1j|6 


Since every element of the s, column is less than or equal to zero, there are no 
candidates for the minimum ratio test, hence this LP is unbounded. The LP 
does not have any optimal solution. 


(b) We have the following tableau: 


The LP has only one optimal solution. 
(c) We have the following tableau: 


Xi X2 Sı 


S2 53 


rhs 


0017 0 0 


10 


103 
0 1 4 3/20 
001 


-1 0 


1 


1 


2 
2 


6 


There is a nonbasic variable (s,) whose coefficient in the zeroth row is zero, so 
this LP has many optimal solutions. 


10.16 (a) The basic variables are x,,x,, and xç. For x, to be basic, a, = 1 and a, = 0. For 


x; to be basic, c, = 0, a, = 0, and a, = 1. 


(b) Setting b > 0 makes the LP feasible. Setting c} > 0, c, > 0, and c} > 0 makes the 


LP optimal. 


(c) Setting b > 0, there are three variables we can introduce into the basis to obtain 


alternative optimal solutions: 
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(i) Variable x,. To do this, at least one of a,,a, must be greater than zero to 
pivot into Column 2. 
(ii) Variable x,. To do this, c} = 0 and a; > 0 to pivot into Column 4. 
(iii) Variable x,. To do this, c, = 0 and a, > 0 to pivot into Column 5. 


10.17 After introducing slack variables, we obtain the following standard form problem: 


10.18 


10.19 


10.20 


max Z=5x,- X, 


s.t. X, — 3X, + X3 = 1, 
x, — 4x, +X, = 3, 
X> XX, Xz, X 20. 


We form the initial tableau: 


rhs} xX, xX, X, X4 

0; 5 -1 0 0 

x,=| 1/-3 1 1 0 

x =| 1 -1 0 1 

The tableau has a nonbasic variable x, that could enter and improve the value of z, 

but there are no candidates for the minimum ratio test. Suppose x, entered the 
basis. The equations representing the constraints are 


x, =1+ 3x, 
X,=1+4+ 4%. 


As x, increases, both x, and x, stay positive. Thus we could keep increasing x, 
(improving z) without ever encountering infeasibility. Hence the LP is unbounded. 


We can easily see that x* and y* are feasible in the primal and dual problems, 
respectively. One can also easily see that b'y* = 19 and c'x* = 19. Based on the 
strong duality property (Theorem 10.9), since b'y* = c'x*, we conclude that x* 
and y* are are optimal in the primal and dual problems, respectively, and their 
optimal value is 19. 


The problem max y; +y, subject to yı — y, < —1, -y, +y, < —1, and y,,y, > 0, 
and its dual problem are a pair with such a property. 


(a) The dual problem is: 
max w = 4y, + 9y, + 5y3 


s.t. Yit 2y, 2 5, 
Yit 3y- Yy; Z 3, 
Yı— Y+ 3y; < -2, 
Yo Y» y32 0. 


(b) (i) The optimal solution to the primal problem is x = (0; 3.5; 1.5). The optimal 
value to the primal problem is 7.5. 
(ii) The optimal solution to the dual problem is y = (0; 1.25; —7.5). The optimal 
value to the dual problem is 7.5. 
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10.21 (a): 


function [x, optimal_cost, iters, Running time] = rsm (A, b, c, 
N, M) 


tl=cputime; 


c=c’'; 

[m n] = size(A); 
bfs=[zeros(1,n-m)’;b] 
B_indices = find(bfs) 
N_indices = find(ones 


(1,n) - abs(sign(bfs))'); 
rsm_nnz = zeros (5000,2); 


% disp(’Please determine the way of choosing the entering 
varialble as follows: '); 
% fprintf(1,’Input 1 if you want to choose the variable with the 
smallest reduced cost to enter the basis.\n’); 
fprintf(1,’Or input 0 if you want to choose the variable that 
first gives a negative reduced cost to enter the basis.\n’); 
EN = inpute(*’ j} 


ole 


% disp(’Please determine the way of choosing the leaving 
varialble as follows: '); 
fprintf(1,’Input "1" if you want to choose the smallest index 
rule.\n’); 
% fprintf(1,’Or input "0" if you want to choose the 
lexicographic. \n’) ; 
z M = input(’ ty; 


ole 


iters=0; 
while 1== 
iters=iters+1; 


Binv = inv(A(:,B indices) ); 


rsm_nnz(iters,1) = nnz(A(:,B indices) ) ; 
rsm_nnz(iters,2) = nnz(Binv); 


d = Binv * b; 


if N == 
c_tilde = zeros(1,n)j; 
c_tilde(:,N_indices) = c(:,N_indices) - c(:,B_ indices) * 
Binv * A(:,N_indices) ; 
[cj j]=min(c_tilde) ; 


if cj >= 0 
x = zeros(n,1); 
=) 


x(B indices, = ds 
optimal_cost C*x; 
X=x!; 
break; 
end; 
end 
if N == 
c tilde = zeros(1,n); 


for k=1:length(N_indices) 
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(b) 


c_tilde(:,N_indices(k)) = c(:,N_indices(k)) - c(:, 
B_ indices) * Binv * A(:,N_indices(k)); 
cj = c_ tilde(:,N_indices(k)) ; 
if cj < 0 
j = N_indices(k) ; 
break 
end; 
end; 
Lf cj S=:0 
x = zeros(n,1); 
x(B_indices,:) = d; 
optimal_cost = c*x; 
xX=x!; 
break; 
end; 
end 
u = Binv * A(:,j); 
mn = inf; 
qag 
zz = find (u > 0)’; 
if (length(zz) == 0) 
x='The LP is unbounded’ ; 
optimal_cost=’The LP is unbounded’; 
break 
else 
[yy, ii] = min (d(zz)./ u (2z)); 
i = zz(ii(l)); 
k = B_ indices (i); 
B_indices(i) = j; 
N_indices(j == N_indices) = k; 
end; 
end; 
t2=cputime; 
Running time=t2-t1; 
function [x, optimal_cost, iters, Running time] = tsm (A, b, c,N 


,M) 
tl=cputime; 


c=c'; 

[m n] = size(A); 
bfs=[zeros(1,n-m)’;b]; 
B indices = find(bfs) ; 


2 


varialble as follows: '); 


g 


o 


% disp('Please determine the way of choosing the entering 


with the smallest index with a negative reduced cost to 


enter the basis.\n’); 
N = inpute Iy 


o 


oe 


% fprintf£(1,’Input "1" if you want to choose the variable with 
the smallest reduced cost to enter the basis.\n’); 
fprintf(1,’Or input "0" if you want to choose the variable 
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% disp(’Please determine the way of choosing the leaving 
varialble as follows: '); 
fprintf(1,’Input "1" if you want to choose the smallest index 
rule.\n'); 
% fprintf£(1,’Or input "0" if you want to choose the 
lexicographic. \n’) ; 
M = input" *); 


ole 


Binv = inv(A(:,B indices) ) ; 

x_B=Binv * b; 

c_ tilde = c - c(:,B_ indices) * Binv * A; 

T_1=[- c(:,B_indices)*x_B, c tilde; x B, Binv * A]; 


m=m-1; 
n=n-1; 


iters=0; 
while 1== 
iters=iters+1; 


if N== 
y=find(T(1,:)<0); 
if length (y) >0 
jey(1)-1; 
else 
x=zeros (1,n); 
for k=2:n+1 


end 
optimal_cost = -T(1,1); 
break 
end 
end 


if N== 
[x_j j]=min(T(1,:)); 
if x_j<0 
j=j-1; 
else 
x=zeros(1,n); 
for k=2:n+1 


end 
end 
optimal_cost = -T(1,1); 
break 
end 
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end 


u = Binv * A(:,j); 

zz = find (u > 0)'; 

if (length(zz) == 0) 
x='The LP is unbounded’; 
optimal_cost=’The LP is unbounded’; 
break 

end 


dind = find(T(2:end,j+1)>0); 


if M== 

[thetast,1] = min( T(1l+dind,1)./ T(1+dind,j+1)); 
l=dind (1); 

end 

if M== 
T(dind+1,:)=T(dind+1,:) ./repmat (T(dind+1,j+1),1,n+1); 


[Ts, sind] =sortrows (T(dind+1,:))j; 
l=dind(sind(1)); 
thetast = T(14+1,1)/T(1+1,j+1); 


end 
T(14+1,:)=T(14+1,:)/T(14+1,5+1); 
for i=setdiff( (1:m+1), 1+1) 
T(i,:) = T(i,:) - T(i,j+1)*T(14+1,:); 
end 


for k=2:n+1 


g=find(T(:,k)}+; 

if length(z)==1 && T(z(1),k) ~= 1 
T(z(1),:)=T(z(1),:)/T(2(1),k); 

end 


end 


if norm(T - T_1)<1.0e-2 
disp(’The simplex method cycles’) ,break 
end 


end 


t2=cputime; 
Running _time=t2-t1; 


10.22 Note that from the first three equations in (10.21) we have that 
(Ax)'y — b'yr 


—x'Aly —x's+xler 


0, 
0, 
—ce'xt+b'yr—xr = 0. 


Now we get the desired equation by adding the above three equations. 


11.1 


11.2 


11.3 
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For eachi = 1,2,...,k, itis known that the matrix M ; = 0 if and only if every prin- 
ciple minor of M; is nonnegative. Since 


ta -1 0 sy 0 . 
0 ol... 0 
det (r,A; - 1) = : ee . : = oa —). 
: : i : j 
0 O n. tåg- l 


It follows that M; > 0 if and only if Mj- 1) > 0, for all s=1,2,...,k 
and det (M,) > 0. Thus, M; > 0 if and only if TjÀij > 1, for all j=1,2,...,k and 


det (M) > 0. 
Notice that 
k k 
det M) = ( (Tii = o) (tivi + d, _x'x) = yu. 
jel j=l 


This means the inequality det (M) > 0 strictly holds for each i < j < k such that 
TjÀij = 1. Hence, det M) > 0 ifand only if 
(wi + d, -¥"X) - 1's, = (tw; +d,-¥ BH — Y, WGAy-)>0. 


TjAy>1 


Therefore, M; > 0 if and only if T;Anin (H) = 1 and d, > Xx- Tv, + 1's,. 
Let p € [1, co]. We first prove that Py C pr. Let x = (%);X) E P”, we show that 
xE pe by verifying that x'y > 0 for any y € Py. So lety = (yo; y) € Py, then 


X'Y = Xp Yo + H'P > |lRlgllHllp + ¥'H > [eG] + > 0, 


where the first inequality follows from the fact that x € P} and y € Pj, and the 
second one from Hölder’s inequality. Thus, P7 € ae 

Now we show pr C Pz. Lety = (3) E Pp" we need to show that y € P3- This 
is trivial if p = 0 or p = œ. If # Oand1 < p < œ, let u £ (y,P/4; y,P/4; ... ; Yp 12/9) 
and consider x Ê (||u|| p> 7U) E Py. Then by using Holder's inequality, where the 
equality is attained, we obtain 


0 <x'y = |lu], Yo — Ud = llull, Yo ~ Ul ILFllg = llull, Wo ~ ll). 


This implies that yy > ||y||,, and therefore means thaty € P}. Thus, Pe C Pj. The 
proof is complete. 


We first show that Kmr C (Ki)*. Letx = (%3X) € Rie we need to show that 


x € (K? )*. For any y = (%;) € K7, we have 
XoYo + x'y 

||(M-*)"X|||| MF] +9 

|km- M5] +9 

= |x'p| +x'p > 0, 


xly 


IV Iv Il 
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where we used the assumptions that x € K uor and y € Ký to obtain the first 
inequality, and we used Cauchy-Schwarz inequality to obtain the second inequal- 


ity. This means that x € (K’j,)* and hence K m-or Cu 


To prove the reverse inclusion, let y € ae ae we need to show that y € Kua 
which is trivial if p = 0. If p # 0, let x ê (\|Myll; yp) € Ky Then, we have 


XVo + X'P 
YollMYll -9'9 
= YollMFl| — ¥'M™(M™)'Y = yolLMFll — IMIM -DFI 
where we used Cauchy-Schwarz inequality, where the equality is attained, to 


obtain the last equality. Since x belongs to the elliptic cone Kj, and y belongs to 
its dual, it follows that 


0 <x'y = ||MYlI(¥o — II) FID. 
As y # 0, this implies that y, > ||(M~!)'j||. That is, y € K” and hence (Kr )* C 


(M1) 
ji ; 
K M-T The proof is complete. 


xly 


11.4 To prove item (c) of Lemma 11.8, note that 


_\T 
x*'st = (= + wAx) (s + aAs) 


I 
xI, 


T —T —T —T 
s+a(Ax s+x'As) +a Ax As 


's + ia trace (owe — X o s) 


I 
xI, 


I 
xI, 


s+ aop trace (e) — ia trace (Xe s) 


I 
xI, 


P a 
St+aour-—ax s 


I 
xI, 


s+ taox' s- ax's = (1 — a(ı — 2)) x's, 


where the third equality follows from items (a) and (b) of Lemma 11.8. 


11.5 Givena E R, using item (c) of Lemma 11.8, we have 


x(a)'s(a) = (1 — a +a0)¥ s, andhence p(a) =(1—-a@+a0)p. 
Thus, we get 


V(a) 


x(a) o s(a) — u(a)e 
= (¥+ aAx) ° (s+ aAs) — (1 — a + ao)ue 


l 


-h h 
—_— la 7 N 
5 Q- a)@ ° s- pe) + a(€ e s — ope) + a(¥ e As + Axes) 
+a? Axe As 
= (1—a)(€os— pe) + a? Ax As. 


This completes the proof. 
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11.6 By the last equation of System (11.15) and from the operator commutativity, we 
have 


h=*oAs+ Axes =o As+pAxexX' + (Axvex')o(¥os— pe). 
It immediately follows that 


al > eas we, JE Bees 


IV 


|E. As+ wax ox '|| - 06, 


= fim ast + [ote ox ff — 05, a6 


= 2 2 2 2 
= 1/62 + 62 — 05, > (1 — 6)4/62 + 62, 


where the second inequality follows from the assumption that ||x © s — yel| < On, 
and the first equality follows from (11.2) and the fact that 


trace (Œ o AS) o (Ax -x")) 


=— =i 
trace (As ° Ax) = Ax As, 


(Œe As)’ (Axe) 


which is essentially zero due to item (a) of Lemma 11.8. 

The right-hand side inequality in (11.18) follows by noting that (8, — 6,)? > 0, and 
the left-hand side inequality in (11.18) follows from the last inequality in (A.6). The 
proof is complete. 


11.7 This is an implementation exercise. As a sample answer, see (Alzalg, 2018, 
Example 7.1). 


11.8 The homogeneous model for the pair (11.29) and (11.30) is as follows: 


WX — hoT = 9, 
B;Xo + W,X;, — hgt = 0,k = 1,2,...,K, 
K 
—Wy'Y - ye + TC, — So = 0, 
k=1 
-W,'y, + te, — 8, = 0,k = 1,2,...,K, (A.7) 


K K 
Vay - vex, —K=0, 
k=0 k=0 

x, = 0,5, > 0,k =0,1,...,K, 
tT>0,x >20. 
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11.9 The search direction system corresponding to (A.7) is defined by the following 
system: 


where 


WAX, —h At = nr o 


ByAxy + WAX, — h,AT = Nro k = 1,2, ...,K, 
K 

-W,' Ayo — YB, AY, + Arey — Asy = nao; 
k=l 


-W, Ay, + AT Cp — As, = nrg, k =1,2,...,K, 

K K 

Va AY: - Ye." Ax, — AK =nr,, 

k=0 k=0 

KkAt+tAk =yyu-Tk, 

AXp ° So + Xo ° ASy = Y Heg — Xo ° Sy; 

AX, ° Sp +X, ° AS, = ye, — Xy ° Sp k =1,2,...,K, 


Py = hot — WoXo; 
A . 
Fok = hyt — ByXy — Wi Xq; 
K 
A yT T ; 
Tao = Wo Yo + ÈB, Yk + So — TCo; 
k=l 
A yT : 
Fak = Wk Yk + Sk — TC; 


ll 


K K 
T 
g 4 k- DIY. + 2 
= x,'8, + 
á ae: k Šk r); 


I> 


and y and y are two parameters. 


12.1 To prove item (c) of Lemma 12.6, note that 


where 


12.2 Given 


Xt e St = (X +aAX) o (S + aAS) 
= XeS+a(AXeS+XeAS)+a*AXe AS 
= X e S + 5a trace (oul — XS) 
= XeS+ acp trace (I) — ia trace (X o» S) 
= XeS+aour-—axeS 
= XoS+ laoX eS—aXeS=(1-a(1-)) Xes, 


the third equality follows from items (a) and (b) of Lemma 12.6. 


a E R, using item (c) of Lemma 12.6, we have 


X(a)e S(a) = (1 — a + ao)X e S, and hence u(a)= (1 — a + ao)n. 


12.3 


12.4 
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Thus, we get 


V(a) = X(a) o s(a) — wall 
= (X + aAX)e(S + aAS) — (1 — a + ac)uI 
-H H 
nmm ma 
= 1-aX ° S— uI) + a(X ° S — onl) + a(X ° AS + AX ° S) 
+a? AX oAS 
= (1 — a)(X o S — uI) +a? AX ° AS. 


This completes the proof. 


Using the last equation of system (12.18) and the commutativity property, we have 
H =X o AS + AX 0 S = X o AS + HAX eX! + (AX eX!) o (XS — pl). 
It immediately follows that 


|Hllp > ||X ° AS + p AX eX], - JAXX] XS — alll 
> ||X ° AS + u AX ° X|; — 05, 


A.8 
= y IXASIIÈ + llu AXX-||} — 06, am 
= 4/8 + 62 — 06, > (1- 8)4/62 + 82, 


where the second inequality follows from the assumption that ||XS — „Ill, < 04, 
and the first equality follows from (12.2) and the fact that 


(X ° AS) o (AX ° X~!) 


trace ((X ° AS) ° (AX ° X~1)) 
trace (AS o AX) = AX e AS, 


which is essentially zero due to item (a) of Lemma 12.6. 

The right-hand side inequality in (12.20) follows by noting that (6, — 6,)* > 0, and 
the left-hand side inequality in (12.20) follows from the last inequality in (A.8). The 
proof is complete. 


This is an implementation exercise. As a sample answer, see (Touil et al., 2017, 
Example 2). 


The homogeneous model for the pair (PISDP) and (DISDP) is as follows (see also 
Potra and Sheng (1998), Jin et al. (2012)): 


AX —br 0, 

—A*y -S +Cr = 0, 

-Ce X +b'y =K = 0, 
X > 0, (A.9) 

S = 0, 

T > 0, 

k > 0. 
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12.6 The search direction system corresponding to (A.9) is defined by the following sys- 
tem (see also Potra and Sheng (1998), Jin et al. (2012)): 


AAX —bAt = NF y 
— A*Ay -AS +CAtr = Ry, 
-Ce AX + b'Ay =- AK =n", 
KAT + tAk = YH H-TK, 
H AXS) + H (XAS) = ul — H (XS), 


(A.10) 


where #,(-) is the symmetrization operator Hp : R’™" ——> S” defined in (12.5), 
n and y are two parameters, and 


4 br — AX, 

4 A*y+S-— 1C, 
2 CeX-b'yt+x, 
X eS +7k), 


nm | 
a 
l> Il 


ll 


I> 


Algorithm A.3: Generic homogeneous self-dual algorithm for SDP 
Input: Data in Problems (PISDP) and (DISDP)(X, y, S, t, x) Ê (I, 0, I, 1, 1) 
Output: An approximate optimal solution to Problem (PISDP) 

1: while a stopping criterion is not satisfied do 

2: | choose yn, y 

3: | compute the solution (AX, Ay, AS, At, Ax) of the linear system (A.10) 
4: | compute a step length @ so that 

X+0AX > 0 

S+60AS > 0 

t+ 0At>0 

k+0Ak>0 

5: | set the new iterate according to 

(X, y, S,T,K) 4 (X, y, S, T, K) + O(AX, Ay, AS, At, Ax) 
6: end 


12.7 We state the generic homogeneous algorithm for solving the pair (PISDP) and 
(DISDP) in Algorithm A.3 (see also (Jin et al., 2012, Algorithm 1)). 


12.8 Under Assumptions 12.2 and 12.1, if the pair (PISDP) and (DISDP) has a solution 
(X*, y*, S*), then Algorithm A.3 finds an e-approximate solution in at most 


o (vnin (trace (X* + S*) (*))) 


iteration (see also (Potra and Sheng, 1998, Theorems 5.2 and 6.2)). 
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